Physics-informed deep learning for fringe pattern analysis

Wei Yin; Yuxuan Che; Xinsheng Li; Mingyu Li; Yan Hu; Shijie Feng; Edmund Y. Lam; Qian Chen; Chao Zuo

doi:10.29026/oea.2024.230034

Abstract

Recently, deep learning has yielded transformative success across optics and photonics, especially in optical metrology. Deep neural networks (DNNs) with a fully convolutional architecture (e.g., U-Net and its derivatives) have been widely implemented in an end-to-end manner to accomplish various optical metrology tasks, such as fringe denoising, phase unwrapping, and fringe analysis. However, the task of training a DNN to accurately identify an image-to-image transform from massive input and output data pairs seems at best na?ve, as the physical laws governing the image formation or other domain expertise pertaining to the measurement have not yet been fully exploited in current deep learning practice. To this end, we introduce a physics-informed deep learning method for fringe pattern analysis (PI-FPA) to overcome this limit by integrating a lightweight DNN with a learning-enhanced Fourier transform profilometry (LeFTP) module. By parameterizing conventional phase retrieval methods, the LeFTP module embeds the prior knowledge in the network structure and the loss function to directly provide reliable phase results for new types of samples, while circumventing the requirement of collecting a large amount of high-quality data in supervised learning methods. Guided by the initial phase from LeFTP, the phase recovery ability of the lightweight DNN is enhanced to further improve the phase accuracy at a low computational cost compared with existing end-to-end networks. Experimental results demonstrate that PI-FPA enables more accurate and computationally efficient single-shot phase retrieval, exhibiting its excellent generalization to various unseen objects during training. The proposed PI-FPA presents that challenging issues in optical metrology can be potentially overcome through the synergy of physics-priors-based traditional tools and data-driven learning approaches, opening new avenues to achieve fast and accurate single-shot 3D imaging.

Introduction

Optical metrology, as a general-purpose metrology technique that uses light as information carriers for non-contact and non-destructive measurement¹, is fundamental to manufacturing, basic research, and engineering applications. With the invention of the laser² and charge-coupled device (CCD)³, many optical metrology methods and instruments are employed in state-of-the-art manufacturing processes, precision positioning, and quality assessment because of their advantages in terms of accuracy, sensitivity, repeatability, and speed. In optical metrology, based on physical models of the image formation, the observed measurements (e.g., deformed fringe/speckle images) can be transformed into the desired physical properties of the objects (the profile, distance, strain, etc.). For many optical measurement techniques such as interferometry⁴, digital holography⁵, and fringe projection profilometry (FPP)^{6, 7}, the accuracy and efficiency of phase retrieval from the recorded fringe images are essential to reconstruct various underlying quantities dynamically. The most efficient method for phase measurement is recovering the phase distribution from a single fringe image, but as a typical case in optical metrology, it is an ill-posed inverse problem. The spatial phase-demodulation (SPD) methods can achieve single-frame fringe analysis by imposing some prior assumptions on the recovered phase (spatially smooth, limited spectral extension, piecewise constant, etc.)^8-10, but at the cost of accuracy and resolution. Since optical metrology experiments are generally carried out in highly customized systems and stringent environments, phase-shifting (PS) methods can provide a deterministic and straightforward solution to the phase retrieval problem by additionally capturing multiple fringe patterns¹¹. PS methods have obvious advantages in terms of speed, accuracy, and repeatability, which have brought up many high-end optical metrology instruments. However, when the optical system is under harsh measurement conditions or the state of the object changes dynamically, PS methods will be severely limited and cannot provide accurate phase recovery results for dynamic measurements. Despite extensive research efforts for decades, how to achieve phase measurement with the highest possible accuracy from the minimum number (preferably single shot) of fringe patterns remains one of the most challenging problems in optical metrology.

With the explosive growth of available data and computing resources, deep learning, as a “data-driven” machine learning technique, has achieved impressive success in numerous fields, such as computer vision and computational imaging¹². Deep learning pervades almost all aspects of optical metrology¹³, and provides solutions to many challenging problems, such as fringe denoising^{14, 15}, fringe analysis¹⁶, and digital holographic reconstruction^17-19. Feng et al.¹⁶ proposed a deep learning method for fringe pattern analysis that establishes an inverse mapping between single-frame fringe and the label phase obtained using 12-step PS method. The trained network can directly estimate the sine and cosine components of fringes, enabling single-shot phase reconstruction with higher accuracy than SPD methods. Recently, phase retrieval methods based on deep learning have been applied to ultrafast 3D imaging (speed up to 20 kHz)²⁰, phase measuring deflectometry²¹, and single-frame absolute 3D measurement²² by adopting diverse deep neural networks (DNNs) with a fully convolutional architecture^{23, 24} or combining the predictions of multiple networks with ensemble learning²⁵. However, these deep learning approaches focus mainly on training a DNN to accurately identify an image-to-image transform from massive input and output data pairs of training datasets without considering the physical laws governing the image formation or other domain expertise pertaining to the measurement. Consequently, the performance of deep learning approaches in solving complex physical problems relies heavily on the underlying statistical characteristics within the dataset. To improve the performance of the network under real experimental conditions, it is necessary to pay a high price for collecting a large amount of high-quality data. In addition, due to the highly customized nature of optical metrology systems, networks trained on one system may not be directly transferable to another system of the same type. Once the new input is different even slightly from the training data, data-driven DNNs may exhibit a poor generalization under diverse measurement conditions, and cannot ensure the interpretability and traceability of their output results. On the contrary, based on accurate physical models of the image formation and its inverse solutions, traditional SPD methods can achieve reliable phase measurements for different types of samples²⁶, but their measurement precision is limited. If the forward physical models of the image formation or traditional solvers of the inverse problem are incorporated into the DNN, it is expected to enhance the performance of deep learning methods while utilizing fewer network parameters. Goy et al.²⁷ proposed a physics-informed deep learning method for phase retrieval at low photon counts that leverages physical priors to convert the raw intensity measurement with noise into an initial estimate of the object, thereby significantly improving the phase reconstruction accuracy by using deep learning. Wang et al.²⁸ demonstrated an unsupervised single-beam phase imaging network to reconstruct the phase of the measured diffraction pattern by integrating a numerically propagated diffraction model. Saba et al.²⁹ proposed a physics-informed neural network for tomographic reconstructions of biological samples, which minimizes the physical loss based on the Helmholtz equation, accurately and quickly retrieving the refractive index distribution from the scattered fields of the sample collected by different illumination directions.

For the limited ability of fringe analysis networks without physics priors, we present a physics-informed deep learning method for fringe pattern analysis (PI-FPA). A learning-enhanced Fourier transform profilometry (LeFTP) module with the prior knowledge of SPD methods is embedded in the DNN to directly provide accurate and reliable phase recovery results for new types of samples, while circumventing the requirement of collecting a large amount of high-quality data in supervised learning methods. The phase results are then refined using a lightweight DNN to further improve the accuracy and computational efficiency of single-shot phase retrieval. Experimental results show that the proposed PI-FPA exhibits superior single-shot fringe analysis performance in speed, accuracy, repeatability, and generalization to various unseen objects during training.

Principle

Phase retrieval from fringe images

Phase retrieval from fringe images is a fundamental task and a representative case among many applications of deep learning in optical metrology. The fringe image $I (x, y)$ is expressed as^{30, 31}

$I (x, y) = A (x, y) + B (x, y) \cos [ϕ (x, y)],$ （1）

where $A (x, y)$ and $B (x, y)$ are the background intensity and the fringe amplitude, and $ϕ (x, y)$ is the phase of the tested object. Retrieving the desired $ϕ (x, y)$ from only one fringe image $I (x, y)$ is an ill-posed inverse problem due to two unknown parts $A (x, y)$ and $B (x, y)$ . In FPP, PS methods¹¹ transform the original ill-posed problem into a well-posed and solvable one by projecting a set of PS patterns to obtain additional observations of the target object:

$I_{n} (x, y) = A (x, y) + B (x, y) \cos [ϕ (x, y) - 2 π n / N],$ （2）

$ϕ (x, y) = \arctan \frac{\sum_{n = 0}^{N - 1} I_{n} (x, y) \sin (2 π n / N)}{\sum_{n = 0}^{N - 1} I_{n} (x, y) \cos (2 π n / N)},$ （3）

where $I_{n} (x, y)$ represents $N$ -step PS images, $ϕ (x, y)$ can be obtained by the least-squares algorithm. However, when the measured object is under harsh measurement conditions, the relative motion between the object and PS patterns will introduce non-negligible errors into phase retrieval results^{32, 33}. Unlike PS methods, SPD methods can realize single-shot phase retrieval using different spatial transform techniques (such as the Fourier transform (FT)⁹ and the windowed Fourier transform¹⁰) under the local smoothness assumption. In Fourier transform profilometry (FTP), the Fourier transform of $I (x, y)$ in Eq. (1) gives

$F_{I} (f_{x}, f_{y}) = F_{A} (f_{x}, f_{y}) + F_{C} (f_{x}, f_{y}) + F_{C^{*}} (f_{x}, f_{y}),$ （4）

$C (x, y) = \frac{1}{2} B (x, y) exp {i 2 π f_{0} x} exp {ϕ_{0} (x, y)},$ （5）

where $F_{A}$ and $F_{C}$ are the Fourier transform of $A (x, y)$ and $C (x, y)$ . $ϕ (x, y)$ is taken as the sum of two independent parts: the object component $ϕ_{0} (x, y)$ and the carrier frequency $2 π f_{0} x$ . Based on the Fourier shift theorem, the zero order $F_{A}$ is separated with ±1 orders $F_{C}$ and $F_{C^{*}}$ , so $F_{C}$ can be extracted by a band-pass filter and converted inversely to the retrieved phase,

$ϕ (x, y) = \arctan \frac{Im {C (x, y)}}{Re {C (x, y)}} .$ （6）

However, when the measured surface contains sharp edges or discontinuities, the support of the zero order and ±1 orders will be extended to cause the spectrum overlapping, precluding high-accuracy phase measurement of complex objects.

Unlike traditional methods that focus on understanding the image formation and solving inverse problems, Feng et al.¹⁶ utilized DNNs to directly estimate the sine and cosine components of $I (x, y)$ for single-shot fringe analysis:

$ϕ (x, y) = \arctan \frac{M (x, y)}{D (x, y)} = \arctan \frac{ρ B (x, y) sin ϕ (x, y)}{ρ B (x, y) cos ϕ (x, y)},$ （7）

where $ρ$ is a constant that depends on phase retrieval methods, e.g., $ρ = 0.5$ for FT methods and $ρ = N / 2$ for $N$ -step PS methods. However, the performance of phase retrieval networks relies heavily on a large amount of high-quality data. Once the new input is different from the training data, the reliability of phase reconstruction results output by data-driven DNNs cannot be guaranteed.

Physics-informed deep learning method for fringe pattern analysis (PI-FPA)

As shown in Fig. 1, different from traditional physics-driven methods (FT methods) and data-driven deep learning approaches (e.g., U-Net and its derivatives) for fringe pattern analysis, the proposed PI-FPA mainly contains a LeFTP module with physics priors and a lightweight network. The LeFTP module, which parameterizes the phase retrieval process of FT methods, utilizes the learnable filters operating in the Fourier transform domain to directly output initial phases in the manner of FTP in Fig. 1(a, b). Physics-driven LeFTP is highly generalizable to provide reliable phase results for various unseen objects during training. The lightweight network refines the initial phase to further improve the phase accuracy at a low computational cost, compared with universal end-to-end image transform networks (U-Net and its derivatives).

Figure 1.Diagrams of the physics-driven method, physics-informed deep learning approach, and data-driven deep learning approach for fringe pattern analysis.

The schematic diagram of the proposed PI-FPA is shown in Fig. 2. First, a net head with a simple convolutional structure is adopted to extract rich low-level features of the input single-frame fringe, which can reduce the effect of the zero order $F_{A}$ of the fringe after training. For the LeFTP module in Fig. 2(c), similar to traditional FT methods, the input tensor is transformed into the Fourier domain through Fourier transform and spectrum centering. Instead of the simple filtering operation of FTP, two learnable filters with multiple channels are utilized to adaptively extract the $+ 1$ order $F_{C}$ closely related to the desired phase. Specifically, a learnable filter $w_{1}$ with $K_{1} \times H_{1} \times W_{1}$ size is applied to weaken the zero order located at the center C1 by weighting each feature of the input spectrum $F_{in}$ pixel by pixel:

Figure 2.Overview of the proposed PI-FPA. (a) PI-FPA including a LeFTP module and a lightweight network. (b) Net head and Net tail. (c) The phase retrieval process of the LeFTP module.

$F_{1}^{K_{1} \times H \times W} = w_{1}^{K_{1} \times H_{1} \times W_{1}} \circ F_{in}^{K_{1} \times H \times W},$ （8）

where $\circ$ is the Hadamard product. Note that the unfiltered high-frequency component is kept to avoid missing details, and the redundant negative Fourier spectrum is removed. Then, a series of filtering operations are implemented to extract delicately the +1 order in various ways using another learnable filter $w_{2}$ with $K_{2} \times H_{2} \times W_{2}$ size:

$F_{2}^{K_{2} \times H \times W} = w_{2}^{K_{2} \times H_{2} \times W_{2}} \cdot \sum_{k = 0}^{K_{1} - 1} F_{1}^{k \times H \times W},$ （9）

where the center of $w_{2}$ is set as C2 estimated by $N$ -step PS. Due to the asymmetry of the spectrum, a large number of reliable and initial phases can be recovered inversely from the filtered spectrum $F_{2}$ according to Eq. (6). Further, to optimize the phase retrieval performance of LeFTP, a priors-based initialization strategy for the filter weights is adopted to facilitate its efficient learning and avoid anchoring in local minima during the training phase by following background-normalized Fourier transform profilometry (BNFTP)³⁴. The filter $w_{1}$ is initialized as an inverse Hanning window for filtering the zero-order component of the input spectrum centered on C1:

$w_{1}^{init} (k, f_{x}, f_{y}) = 1 - \cos \frac{2 π f_{x}}{W_{1}} .$ （10）

In addition, the +1 order of the spectrum centered on C2 is strengthened using another Hanning filter $w_{2}$ :

$w_{2}^{init} (k, f_{x}, f_{y}) = \cos \frac{2 π f_{x}}{H_{2}} \cos \frac{2 π f_{y}}{W_{2}} .$ （11）

At present, mainstream fringe analysis approaches using deep learning exploit end-to-end fully convolutional networks in a naïve manner to build an image-to-image inverse mapping between single-frame fringe and the label phase using massive network parameters. Thanks to robust phase estimation of LeFTP, it not only helps PI-FPA to circumvent the requirement of collecting a large amount of high-quality data in supervised learning methods, but also relieves the burden of phase refinement for lightweight DNNs. The lightweight network, consisting of the context path and the spatial path inspired by BiSeNet^{35, 36} in Fig. 2(a) (see Supplementary information for detailed analysis), is utilized to further improve the phase accuracy at a low computational cost compared with universal end-to-end image transform networks (U-Net and its derivatives). Instead of configuring more channels for higher-level layers as U-Net, the context path aims at collecting the fringe and initial phase features with a large receptive field through fast downsampling and encoding global context information to guide the refined high-level features for learning, while the spatial path captures spatial information encoding rich detail information and outputs low-level features. In the encoder part of the context path, a fast downsampling strategy with several ConvX blocks and the Short-Term Dense Concatenate (STDC) module is used to extract the feature information with scalable receptive field and multi-scale information. In the decoder phase, the attention-based feature refinement (AFR) module and the fast upsampling operation based on bilinear interpolation are utilized to improve the feature resolution progressively. In the spatial path, its encoder part shares the same parameters with the context path, and captures the spatial information encoding rich detailed information and outputs low-level features. The features from the context path and the spatial path are concatenated by Feature Fusion module (FFM), and upsampled to output final phases using the predicted $M (x, y)$ and $D (x, y)$ in Eq. (7). The objective of PI-FPA is to minimize the joint loss of the phase and its Fourier domain:

$L o s s = L o s s_{phase} + L o s s_{Fourier},$ （12）

$L o s s_{phase} = \frac{α_{1} {| Y - Y_{GT} |}^{2} + α_{2} {| N \cdot Y_{LeFTP} - Y_{GT} |}^{2}}{H W},$ （13）

$L o s s_{Fourier} = \frac{β_{1} | F_{Y} - F_{Y_{GT}} | + β_{2} | N \cdot F_{Y_{LeFTP}} - F_{Y_{GT}} |}{H W},$ （14）

where $Y_{GT} = (M_{GT}, D_{GT})$ is the ground truth obtained using $N$ -step PS, $Y = (M, D)$ is the network’s output, $Y_{LeFTP} = (M_{LeFTP}, D_{LeFTP})$ is the LeFTP module’s output, and $F_{Y}$ , $F_{Y_{GT}}$ , and $F_{Y_{LeFTP}}$ are the 2D Discrete Fourier Transform of $Y$ , $Y_{GT}$ , and $Y_{LeFTP}$ .

Experiments

In order to verify the proposed PI-FPA under the scenario of FPP, we built a multi-view structured light system that consisted of a projector (LightCrafter 4500Pro, Texas Instruments) and three cameras (acA640-750um, Basler) (see Supplementary information for detailed analysis). To collect fringe data for training, the projector projects three sets of PS patterns with different periods (including 1, 8, and 64) onto the test objects. The captured 64-period fringe image is the input of PI-FPA, and the label phase is obtained by 12-step PS. In the experiment, we collected the dataset including 1200 image pairs, which are divided into 800 image pairs for training, 200 image pairs for validation, and 200 image pairs for testing. The proposed PI-FPA is implemented using Pytorch framework (Facebook) and is computed on an NVIDIA GeForce RTX2080Ti graphics card. The composite loss function consists of mean square error (MSE) and mean absolute error (MAE) in Eq. (12). The optimizer is Adam, and the training epoch is set as 300.

First, a David plaster was measured to reveal single-shot phase retrieval process of PI-FPA, and FTP, LeFTP, Net head + LeFTP, and U-Net were implemented for comparison. In Fig. 3(a, b), LeFTP makes use of two learnable filters operating in the Fourier domain and reduces the MAE of phase errors by about 18% compared with FTP. By visualizing the filter weights, it demonstrates that LeFTP facilitates adaptive spectrum extraction through learning-enhanced filtering, which provides an interpretable guide for parameter optimization of FTP to improve the phase accuracy. In addition, due to the removal of redundant negative Fourier spectra in LeFTP, the left half of the filter weights is the same as its initial state, which is not updated during network training. To further speed up the LeFTP module, it is optional to cut down the size of two learnable filters in half to reduce the total parameters of the network and improve the inference speed of the network. Further, the Net head in Fig. 3(c), taken as the filtering operation in image pre-processing, is embedded in the front of LeFTP to extract rich low-level fringe features for removing the zero order, further reducing the phase errors by about 40%. It proves that LeFTP is plug-and-play to significantly boost the performance of single-frame fringe pattern analysis.

Method	Time (ms)	RMS (μm)
Method	Time (ms)	Plane	Sphere
3-step PS	5.22×10⁻³	188±29.8	179±19.9
FTP	2.06×10⁻²	77±6.8	81±7.4
U-Net	65.02	56±4.9	59±6.6
PI-FPA	18.78	43±4.1	47±5.1

Table 1. Quantitative analysis results of the moving plane and sphere for different methods.

View all Tables

Figure 3.Comparative results for single-shot fringe pattern analysis of the David model. (a–e) The phase retrieval process, wrapped phases, phase errors, and magnified views of the phase errors using FTP, LeFTP, Net head + LeFTP, U-Net, and PI-FPA.

Different from these methods above, U-Net automatically exploits massive low-level and high-level features to optimize the phase accuracy as shown in Fig. 3(d), but at the cost of computational overhead. Specifically, U-Net needs 3.5 GB of GPU memory to process single-frame fringe and takes a runtime of 65.02 ms on Nvidia RTX 2080Ti. Guided by reliable phase results provided by Net head + LeFTP, in Fig. 3(e), PI-FPA refines the phases through a lightweight DNN, which reduces the GPU memory to 1.5 GB and improves the speed to 53.23 FPS while decreasing the MAE by about 20%. The magnified maps of phase errors in Fig. 3 indicate that the trained PI-FPA is able to reconstruct high-quality phase information for local fine details of objects with complex surfaces. In addition, the results of U-Net and PI-FPA using different amounts of training images are presented in Fig. S5 (see Supplementary information for detailed analysis). Compared with U-Net with 800 training image pairs, PI-FPA reduces the MAE of the phase errors by about 13% while requiring only 400 training image pairs, which demonstrates its good generalization.

To verify the generalization of PI-FPA for complex surfaces, we tested an industrial part, and fringe analysis results using different methods show that the phase errors are smaller in smooth cylindrical regions but larger in sharp edges, while PI-FPA brings better phase quality among these methods as shown in Fig. 4(a, b). Further, we adopted stereo phase unwrapping³⁷ to achieve single-shot 3D imaging in Fig. 4(c) (see Supplementary information for detailed analysis). As the magnified regions in Fig. 4(d), the screw thread of the workpiece, which is relatively rare in the training dataset, causes significant degradation in the performance of U-Net, precluding high-precision reconstruction of complex surfaces. The line profiles in Fig. 4(e) prove that the proposed PI-FPA with physics-driven LeFTP can successfully recover the fine profiles of the threads and provide accurate and physically consistent 3D imaging results to approach the ground truth (GT), even though the network has not seen such experimental data during the training phase.

Figure 4.Comparative fringe analysis results of the industrial part. (a) The industrial part and the phase errors using FTP, U-Net, and PI-FPA. (b) The magnified views of the phase errors. (c) Single-shot 3D imaging results using different methods. (d) The magnified views of (c). (e) The line profiles in (d).

To quantitatively analyze the 3D imaging accuracy of PI-FPA, our system was applied to measuring a dynamic scene at the camera speed of 100 Hz: a ceramic plane and a standard sphere moving along the Z axis, and 3D reconstruction results at different time points are shown in Fig. 5(a). The error distributions of the moving sphere are obtained by sphere fitting at T = 0 s, 0.81 s, and 1.62 s, where major measured errors are less than 100 μm with the RMS of 52.198 μm, 42.112 μm, and 53.295 μm as shown in Fig. 5(b). Similarly, Fig. 5(c) shows the measured RMS of the moving plane is 51.425 μm, 38.922 μm, and 37.183 μm. In Fig. 5(d, e), we further perform temporal precision analysis by collecting long-term data over a 1.62 s period using 3-step PS, FTP, U-Net, and PI-FPA. In Table 1, quantitative analysis results of the moving plane and sphere for different methods show that the measured results obtained by PI-FPA exhibited higher 3D reconstruction accuracy with a lower temporal standard deviation (STD) of 43 $\pm$ 4.1 μm and 47 $\pm$ 5.1 μm.

Figure 5.Precision analysis for a ceramic plane and a standard sphere moving along the Z axis. (a) 3D reconstruction results using PI-FPA at different time points. (b–c) the error distributions of the sphere and plane. (d–e) temporal precision analysis results of the plane and sphere over a 1.62 s period using 3-step PS, FTP, U-Net, and PI-FPA. (f–i) the color-coded 3D reconstruction and the corresponding error distributions of the plane and the standard sphere using different methods at T = 0.81 s.

In Fig. 5(f–i), we additionally provide the measurement results of the moving plane and sphere at T = 0.81 s using different methods. Different from FTP for single-shot phase retrieval, PS methods can realize pixel-by-pixel phase measurements with higher accuracy for complex shapes, but it needs to project at least three fringe patterns to obtain a phase map theoretically. As the most common and efficient case in N-step PS methods, 3-step PS is implemented for comparison. When dynamic scenes are measured, the relative motion between the object and the phase-shifting fringe patterns sequentially projected will cause motion artifacts and thus introduce non-negligible phase errors into the phase map. As a consequence, there are severe measurement errors with the RMS of 196.101 μm and 179.681 μm in the measurement results of 3-step PS in Fig. 5(f). In addition, for real-time 3D measurement based on 3-step PS, the whole procedure of 3D reconstruction is composed of phase retrieval, stereo phase unwrapping, and phase-to-height mapping, which is implemented with a graphics processing unit (GPU)³⁸ and several look-up tables³⁹ to speed up the 3D reconstruction. The 3D imaging speed is determined by the maximum between the image acquisition time and the runtime of 3D reconstruction. The runtime of stereo phase unwrapping³⁷ and phase-to-height mapping for processing the images with the resolution of $640 \times 480$ pixels is less than 5 ms on RTX2080Ti. Since 3-step PS needs to capture three fringe images and its runtime of the phase retrieval is negligible (5.22 $\times 10^{- 3}$ ms) in Table 1, its 3D imaging speed is limited to 33.33 FPS. On the contrary, the single-frame fringe analysis capability of FTP can significantly improve the accuracy and repeatability of fast 3D measurement to reduce the RMS to 75.417 μm and 71.715 μm, while its runtime (2.06 $\times 10^{- 2}$ ms) promotes the speed of 3D measurement to 100 FPS in Fig. 5(g). This result proves that single-frame fringe analysis methods are more suitable for dynamic scene measurement when the target's movement speed is in the same order of magnitude as the 3D imaging speed. Then, in Fig. 5(h), the RMS of the measurement error can be further decreased to 53.361 μm and 60.129 μm thanks to the powerful feature extraction capability of U-Net, but at the cost of lower inference speed (65.02 ms), precluding real-time 3D measurement. Finally, benefiting from the proposed LeFTP module and the lightweight DNN, PI-FPA takes a runtime of 18.78 ms to achieve fast single-shot phase reconstruction with higher accuracy in Fig. 5(i). However, PI-FPA only retrieves the phase of the first in the three-step PS images and reduces the 3D imaging speed to 33.33 FPS. 3D measurement results in Fig. 5 confirm that PI-FPA, whether measuring the moving plane or sphere, achieves successfully single-shot 3D shape measurement with higher accuracy and good repeatability for multiple moving objects simultaneously. The whole 3D measurement results can refer to Supplementary Video S1.

Last, to further demonstrate the advantages of PI-FPA, we applied our single-shot 3D imaging system to 360-degree reconstruction of a workpiece model and non-rigid dynamic face measurement as shown in Fig. 6 and Supplementary Video S2–S3. Fig. 6(a, b) show the captured fringe images of the rotated workpiece and non-rigid dynamic face at different time points and the corresponding color-coded 3D reconstruction results using different methods. For the rotated workpiece, the highlighted regions in Fig. 6(a) show that 3-step PS cannot recover the fine shapes of smooth surfaces due to the phase errors introduced by motion artifacts. For single-frame fringe analysis, FTP is suitable for dynamic 3D measurement, but yields coarse 3D results with low quality in terms of accuracy and resolution due to the spectrum overlapping. U-Net can further improve the quality of 3D reconstruction, but it cannot reliably retrieve the phase of the object with metal materials which is relatively rare in the training dataset, precluding the recovery of fine surfaces. This experiment demonstrates that the proposed PI-FPA can be applied for high-quality and efficient 3D modeling of complex structure parts as shown in Fig. 6(c). Similarly, for non-rigid dynamic face, there are inevitably a large amount of ripple-like measurement errors in 3D results of 3-step PS due to motion artifacts. And then, FTP is performed to significantly reduce measurement errors, but is unable to recover high-quality local details of the face. Due to the smooth and diffuse properties of faces, both PI-FPA and U-Net provide acceptable 3D face measurement results. Because of the lack of 3D label data for the tested face, it cannot identify precisely which of these two results is better, but there are slight differences in some local details, such as the left cheek and the tip of the nose in Fig. 6(b). In the whole measuring procedure, the reconstructed dynamic face at different time points verified the reliability of PI-FPA to perform fast 3D shape measurement with high completeness as well as see Supplementary Video S3. These results suggest that PI-FPA is a promising tool for fast 3D measurement and reverse modeling with high quality for objects with complex shapes.

Figure 6.Fast 3D measurement results using different fringe pattern analysis methods. (a) The representative fringe images at different time points and the corresponding color-coded 3D reconstructions results for the rotated workpiece model using 3-step PS, FTP, U-Net, and PI-FPA. (b) The representative fringe images at different time points and the corresponding color-coded 3D reconstructions results for non-rigid dynamic face using 3-step PS, FTP, U-Net, and PI-FPA. (c) 360-degree 3D reconstruction of the workpiece model using PI-FPA. (d) 3D measurement results of non-rigid dynamic face using PI-FPA.

Conclusions and discussion

In summary, we have demonstrated a physics-informed deep learning method for fringe pattern analysis (PI-FPA) that is able to achieve accurate and computationally efficient single-shot phase reconstruction and exhibits strong generalization capability to new types of samples. By introducing the LeFTP module with the prior knowledge of traditional phase demodulation methods, PI-FPA circumvents the requirement of collecting a large amount of high-quality data, while overcoming the degradation of reconstruction quality for rare samples or structures in supervised learning methods. Utilizing reliable phase results from LeFTP as the network input, PI-FPA strengthens the ability of the lightweight DNN to further improve the phase recovery accuracy at a low computational cost compared with existing end-to-end networks. The effectiveness of PI-FPA has been verified by several experiments for measuring various types of static and dynamic scenes. The single-shot phase retrieval results of the David plaster confirmed that PI-FPA can reconstruct high-quality phase information for objects with complex surfaces, while also achieving an improvement of 3.46× in its network inference speed compared with U-Net. By adopting stereo phase unwrapping, PI-FPA has the capability of single-frame 3D imaging to successfully recover the fine profiles of the industrial part with the threads, exhibiting its good generalization to rare samples never seen by the network. Temporal precision analysis results verified the high accuracy and excellent repeatability of PI-FPA for measuring multiple moving objects simultaneously. Finally, 360-degree reconstruction of a workpiece model and non-rigid dynamic face measurement revealed the applicability of PI-FPA for fast 3D measurement with high quality for objects with complex shapes and different materials. In the future, the performance of PI-FPA for phase retrieval from various types of fringe images will be investigated. We wish that PI-FPA can be applicable to other fringe analysis applications in optical metrology, further pushing the limits of fringe pattern analysis in speed, accuracy, repeatability, and generalization.

References

[1] KJ Gåsvik. Optical Metrology(2002).

[2] JC Wyant, K Creath. Recent advances in interferometric optical testing. Laser Focus, 21, 118-132(1985).

[3] R Kulkarni, P Rastogi. Optical measurement techniques-A push for digitization. Opt Lasers Eng, 87, 1-17(2016).

[4] P Hariharan. Basics of Interferometry(2010).

[5] U Schnars, C Falldorf, J Watson et al. Digital Holography and Wavefront Sensing(2015).

[6] SS Gorthi, P Rastogi. Fringe projection techniques: whither we are?. Opt Lasers Eng, 48, 133-140(2010).

[7] J Geng. Structured-light 3D surface imaging: a tutorial. Adv Opt Photon, 3, 128-160(2011).

[8] M Servin, JA Quiroga, JM Padilla. Fringe Pattern Analysis for Optical Metrology: Theory, Algorithms, and Applications(2014).

[9] XY Su, WJ Chen. Fourier transform profilometry: a review. Opt Lasers Eng, 35, 263-284(2001).

[10] Q Kemao. Windowed fourier transform for fringe pattern analysis. Appl Opt, 43, 2695-2702(2004).

[11] C Zuo, SJ Feng, L Hiang et al. Phase shifting algorithms for fringe projection profilometry: a review. Opt Lasers Eng, 109, 23-59(2018).

[12] G Barbastathis, A Ozcan, G Situ. On the use of deep learning for computational imaging. Optica, 6, 921-943(2019).

[13] C Zuo, JM Qian, SJ Feng et al. Deep learning in optical metrology: a review. Light Sci Appl, 11, 39(2022).

[14] KT Yan, YJ Yi, CT Huang et al. Fringe pattern denoising based on deep learning. Opt Commun, 437, 148-152(2019).

[15] R Kulkarni, P Rastogi. Fringe denoising algorithms: a review. Opt Lasers Eng, 135, 106190(2020).

[16] SJ Feng, Q Chen, GH Gu et al. Fringe pattern analysis using deep learning. Adv Photon, 1, 025001(2019).

[17] ZB Ren, ZM Xu, EYM Lam. End-to-end deep learning framework for digital holographic reconstruction. Adv Photon, 1, 016004(2019).

[18] Y Rivenson, YB Zhang, H Günaydın et al. Phase recovery and holographic image reconstruction using deep learning in neural networks. Light Sci Appl, 7, 17141(2018).

[19] KX Liu, JC Wu, ZH He et al. 4K-DMDNet: diffraction model-driven network for 4K computer-generated holography. Opto-Electron Adv, 6, 220135(2023).

[20] SJ Feng, C Zuo, W Yin et al. Micro deep learning profilometry for high-speed 3D surface imaging. Opt Lasers Eng, 121, 416-427(2019).

[21] G Qiao, YY Huang, YP Song et al. A single-shot phase retrieval method for phase measuring deflectometry based on deep learning. Opt Commun, 476, 126303(2020).

[22] YX Li, JM Qian, SJ Feng et al. Deep-learning-enabled dual-frequency composite fringe projection profilometry for single-shot absolute 3D shape measurement. Opto-Electron Adv, 5, 210021(2022).

[23] T Yang, ZZ Zhang, HH Li et al. Single-shot phase extraction for fringe projection profilometry using deep convolutional generative adversarial network. Meas Sci Technol, 32, 015007(2020).

[24] W Yin, JX Zhong, SJ Feng et al. Composite deep learning framework for absolute 3D shape measurement based on single fringe phase retrieval and speckle correlation. J Phys Photon, 2, 045009(2020).

[25] SJ Feng, YL Xiao, W Yin et al. Fringe-pattern analysis with ensemble deep learning. Adv Photon Nexus, 2, 036010(2023).

[26] W Osten. What optical metrology can do for experimental me chanics. Appl Mech Mater, 70, 1-20(2011).

[27] A Goy, K Arthur, S Li et al. Low photon count phase retrieval using deep learning. Phys Rev Lett, 121, 243902(2018).

[28] F Wang, YM Bian, HC Wang et al. Phase imaging with an untrained neural network. Light Sci Appl, 9, 77(2020).

[29] A Saba, C Gigli, AB Ayoub et al. Physics-informed neural networks for diffraction tomography. Adv Photon, 4, 066001(2022).

[30] GT Reid. Automatic fringe pattern analysis: a review. Opt Lasers Eng, 7, 37-68(1986–1987).