This study deals with the development of a new fiber-optic based sensor probe for in-process measurement of the machined surface roughness in manufacturing system. A simulation methodology for the determination of the relationship between the performance of the sensor probe and its structure parameters was established, which is used for the probe's optimization design. The usefulness of developed simulation method was verified through the experimental study.