With the rapid development of high-end equipment manufacturing and the widespread adoption of photoelectric detection technology, laser interferometric displacement measurement technology [1] has emerged as a crucial tool in the realm of ultra-precision machining. This technology offers distinct advantages, such as high precision, non-contact measurement, and traceability [2]. Traditional mirror-based laser interferometers, while highly precise, often exhibit large physical footprints and demand stringent measurement environments. In contrast, embedded fiber-optic micro-probe laser interferometers (FMIs) characterized by their small size, high integration, robust environmental adaptability, and online measurement capabilities, play an important role in confined measurement environments and subnanometer resolution measurement [3–5].

The performance of FMI is largely limited by demodulation technology. As a mature technology, white light interferometers, utilizing broadband light sources for cavity length measurement, can achieve subnanometer resolution [6]. However, this remarkable resolution is primarily suitable for quasi-static measurements and may not perform optimally in high-speed measurements. Consequently, the quadrature intensity demodulation technique [7,8] has gained popularity for fast measurements, yet it is constrained by a limited dynamic range ($\lambda /4$), which brings limitations in practical applications. In addition to the above-mentioned methods, phase-generated carrier (PGC) technology offers advantages including good linearity, broad dynamic range, and high sensitivity, making it a widely adopted signal demodulation technology [9,10].

Typically, the maximum measurement speed of the interferometer is limited by the laser modulation frequency [11]. PGC technology needs to generate a high-frequency carrier phase signal based on the modulated laser frequency and the idle length of the interferometer, so as to be applied to high-speed measurement. He et al. proposed an inverse tangent differential self-crossing algorithm (PGC-Arctan-DSM), which attains a signal-to-noise-and-distortion (SINAD) of 60 dB [12]. In 2021, Yan et al. proposed a nonlinear error compensation method based on the PGC-Arctan algorithm to eliminate the nonlinear errors of nanoscale displacement measurements [13]. In 2022, Chen et al. introduced a demodulation scheme that combines PGC-Arctan and PGC-DCM to reduce nonlinear errors from 3.65 nm to 0.10 nm at a displacement of 1 μm/s [14]. The same year, Li et al. presented an enhanced PGC demodulation algorithm based on frequency mixing and division difference [15]. Compared to traditional PGC-DCM and PGC-Arctan algorithms, this method improves the signal-to-noise ratio (SNR) by 26.26 dB and 35.66 dB, respectively. However, all these methods analyze the signal quality of the demodulation results and do not directly establish a relationship between PGC system performance and displacement measurement resolution, thus lacking a basis for subnanometer resolution design. Furthermore, the optimization algorithms for PGC are designed for short-distance and low-speed measurement conditions, and a high-speed PGC demodulation model with subnanometer precision has not been established and realized.

At present, the primary factors affecting PGC demodulation include accompanied optical-intensity modulation (AOIM), phase modulation depth (PMD), and carrier phase delay (CPD), making further enhancement of interferometer demodulation precision challenging. In the existing technology, errors arising from AOIM and PMD have been effectively mitigated [16–20], while errors caused by CPD remain difficult to suppress, and even the displacement signal cannot be demodulated successfully under high-speed and long-distance measurement conditions. To address this, more suitable solutions, such as the elliptic fitting method, have been proposed [21,22]. However, this approach has the drawbacks of substantial computational requirements and slow operation speed and is generally used for off-line measurements. In contrast, the eigenvalue extremum method, based on the elliptic fitting algorithm, enables real-time online measurement. Nevertheless, its limitation lies in its limited sampling rate, making it challenging to extract accurate extremum values under conditions of slow sampling rates. Furthermore, the elliptic fitting algorithm may fail when the Lissajous diagram of the system forms a straight line.

To address the above-mentioned CPD-related issues, Dong et al. [23] proposed a CPD prediction method. However, this method is a static prediction of CPD, which cannot meet the dynamic measurement requirements. In response to this problem, Wang et al. [24] employed the Arctan algorithm in PGC to estimate the real-time carrier phase delay compensation amount, but when the phases to be measured correspond to integer multiples of $\pi$, the demodulation results exhibit discontinuities. Then Xie et al. [25] proposed a CPD extraction method that effectively avoids the above problem, but also brings a new problem, that is, when the CPDs correspond to odd multiples of $\pi /4$, the demodulation results exhibit discontinuities. These jump points significantly affect modulation accuracy, highlighting the need for reliable and effective online CPD compensation under high-speed and large-range measurement conditions.

This paper introduces a resolution equivalent model based on the PGC-Arctan demodulation algorithm with internal modulation. This model effectively represents the PGC demodulation process as a signal acquisition system that includes equivalent error terms. It establishes a relationship between the effective number of bits $N^{*}$ in the equivalent model and the actual precision of the demodulation system. This approach allows for confident error allocation in the design of FMIs, offering valuable guidance for this type of design. Furthermore, the paper conducts a theoretical analysis of phase demodulation errors that occur in traditional PGC algorithms under high-speed and large-range measurement conditions. In such scenarios, issues like characteristic parameter disappearance and system CPD cannot be ignored, as CPD has a significant impact on the calculation results. Therefore, an adaptive compensation algorithm, combining dynamic and static CPD, is designed. Experimental results demonstrate that using this compensation algorithm, measurement errors caused by CPD can be reduced to less than 0.5 nm at a measurement speed of 1.5 m/s, meeting subnanometer level resolution precision requirements.

The subsequent sections of this paper are organized as follows. In Section 2, the whole work of this paper is introduced. In Section 3, an equivalent model for subnanometer precision resolution using the PGC demodulation algorithm under high-speed measurement is established. Furthermore, the relationship between the performance of the demodulation system and the equivalent model parameters is highlighted. Additionally, the influence of CPD, AOIM, and PMD on the demodulation results is analyzed, while offering reasonable solutions based on the underlying generation mechanism. In Section 4, the impact of CPD on the quality of demodulation signals is assessed through simulations, and a high-precision delay dynamic adaptive PGC demodulation algorithm is designed. Finally, the feasibility and advanced nature of this method are validated through comparative experiments.

The overall context of this paper is shown in Fig. 1. A novel resolution-equivalent model is proposed for the first time to evaluate the accuracy of displacement measurement, based on the measurement concept of FMI and the operating principle of the PGC demodulation algorithm. After analyzing the error terms of each link in FMI, aiming at the nonlinear error introduced by the CPD phenomenon under high-speed measurement, a high-precision PGC demodulation algorithm with delay dynamic adaptive regulation is proposed for the first time.

In Fig. 1, the lower-left corner represents the overall actual hardware platform structure containing the optical path and circuitry, and the gray block diagram in the upper-left corner is its equivalent structure. Combining these two structure diagrams, it can be seen that in this paper, the overall structure is equivalent to an analog-to-digital converter (ADC) containing each error term, whose input is the phase to be measured ${\phi }_0$, and the output is the calculated phase ${\phi }^{\prime }$, and based on this equivalence process, a resolution equivalent model is established for evaluating the accuracy of the FMI displacement measurement. From the results of the equivalence, integrate all the errors and noise in the FMI system and consider them as the quantization errors of the equivalent ADC. Put simply, the accuracy of the displacement measurement of FMI can be quantified by the effective number of bits of the equivalent ADC.

Then, analyze the error terms introduced by each error source in the measurement system, such as the error terms $M_{\tau }$ and $A_{\tau }$ introduced by the CPD phenomenon, and establish the phase demodulation error model according to the errors existing in FMI. The phenomenon of PGC not working properly under high-speed measurement is revealed initially, and simultaneously, the CPD phenomenon is found to be the key reason limiting the further improvement of displacement measurement accuracy under high-speed measurement. Hence, a high-precision PGC demodulation algorithm with delay dynamic adaptive regulation is designed. This algorithm is depicted in the blue algorithm block diagram located in the upper-right corner.

Finally, following experimental analysis, the comparison graphs of displacement measurement accuracy between high-speed and low-speed, traditional PGC methods [24,25] and the proposed PGC method are plotted in gray, red, and blue in the lower-right corner. From these three depicted accuracy versus work distance curves, it can be seen that with the increasing measurement speed, traditional PGC methods exhibit curve distortion due to the CPD phenomenon, while the proposed PGC method still has high measurement accuracy at the same high measurement speed, realizing a displacement resolution of 0.1 nm with a standard deviation of less than 0.5 nm at a speed of 1.5 m/s.

Figure 2 illustrates the block diagram of the PGC demodulation algorithm based on FMI. This paper presents an FMI, characterized by its small size, common optical path, and easy integration. The phase to be measured, ${\phi }_0(t)=4\pi n·L_0(t)/\lambda$, is directly proportional to the displacement to be measured, $L_0(t)$, where $n$ is the refractive index and $\lambda$ is the output wavelength of the laser. The input signal passes through the FMI, photodetector, and high-speed analog-to-digital converter (ADC) to create an interference signal, which then enters the PGC demodulation system for signal processing. The PGC algorithm can be regarded as a combination of two lock-in amplifiers and an Arctan algorithm, and the specific demodulation process unfolds as follows: the collected interference signal splits into two channels, and both interference signals are simultaneously multiplied by the first and second harmonic carrier signals at the modulation frequency. They go through low-pass filters (LPFs) and become two orthogonal sine and cosine signals, denoted as $u_1$ and $u_2$, respectively. Subsequently, the two orthogonal signals are divided first and then the inverse tangent operation is performed to obtain the calculated phase, ${\phi }^{\prime }(t)$.

The PGC demodulation algorithm derives the output phase, $\phi (t)=\arctan (u_1/u_2)$. The output error of the system, $\mathrm{\Delta }\phi (t)$, in the form of uncertainty, is evaluated as follows: $$\mathrm{\Delta }\phi (t)=\sqrt{{(\frac{\partial \phi (t)}{\partial u_1}·\mathrm{\Delta }{u}_1)}^2+{(\frac{\partial \phi (t)}{\partial u_2}·\mathrm{\Delta }{u}_2)}^2}=\sqrt{{\left(\frac{u_2}{u_1^2+u_2^2}\right)}^2\mathrm{\Delta }{u}_1^2+{\left(\frac{u_1}{u_1^2+u_2^2}\right)}^2\mathrm{\Delta }{u}_2^2},$$(1)where $\mathrm{\Delta }{u}_1$ and $\mathrm{\Delta }{u}_2$ are the errors of $u_1$ and $u_2$, respectively.

Since $u_1$ and $u_2$, obtained by analog-to-digital conversion within the solution system, originate from the same ADC, the resolution of the two signals is identical. Consequently, $\mathrm{\Delta }{u}_1$ can be assumed to be equal to $\mathrm{\Delta }{u}_2$. Through reasonable design, ignoring the errors introduced by CPD, AOIM, and filters, and when the phase modulation depth $C$ is 2.63 [13], the amplitudes of the two signals, $U_1$ and $U_2$, are also equal. Utilizing the trigonometric function relationship, the output error of the system can be further simplified as follows: $$\mathrm{\Delta }\phi (t)=\frac{\mathrm{\Delta }{u}_1}{U_1}.$$(2)

It can be observed that the final simplified expression of the phase error, $\mathrm{\Delta }\phi (t)$, is equivalent to the dynamic range of the output phase of the system.

According to Fig. 2, the solid green wireframe and solid yellow wireframe on the left correspond to ${\mathrm{ADC}}_1^{*}$ and ${\mathrm{ADC}}_2^{*}$, marked by the dotted line frames of the corresponding colors on the right. Consequently, the entire interferometry and phase demodulation system can be regarded as two identical ADCs with the effective number of bits $N^{*}$, serving to convert analog displacement measurements to digital values. Existing displacement errors $\mathrm{\Delta }L$ in the conversion process can be expressed as follows: $$\mathrm{\Delta }L=\frac{\lambda }{4\pi n}\mathrm{\Delta }\phi (t)=\frac{\lambda }{4\pi n}·\frac{1}{2^{N^{*}}}.$$(3)

The relationship between SINAD and $N^{*}$ of the equivalent model can be expressed as follows: $$N^{*}=\frac{\mathrm{SINAD}[\mathrm{dB}]-1.76[\mathrm{dB}]}{6.02[\mathrm{dB}]}.$$(4)

As shown in Fig. 3, equivalent acquisition bits and SINAD change with displacement measuring resolution can be derived from Eqs. (3) and (4). For instance, in an experiment utilizing a laser with a central wavelength of 1532.8 nm, the equivalent ADC resolution should be around 10.3 bits to achieve an interferometric accuracy of 0.1 nm. Hence, reasonable error distribution is required, offering guidance in the design of interference and calculation systems.

According to the modulation and demodulation principle of FMI, the errors in the optical path are initially analyzed. First, there is a time delay $\tau$ between the laser output from the laser tube and the photodetector. Second, when the driving current modulates the frequency of the laser, it inevitably modulates the light intensity, causing the amplitude of the optical signal to periodically change according to the modulation frequency. These factors introduce nonlinear errors into the demodulation results. At this point, the interference signal $i_m$ [23] can be represented as $$\begin{gathered} i_m=(1+m\cos ({\omega }_0(t-\tau )+{\phi }_m))·(A+B\cos (C\cos ({\omega }_0(t-\tau ))+{\phi }_0(t))) \\ =(1+m\cos ({\omega }_{0}t-{\phi }_{\tau }+{\phi }_m))·(A+B\cos (C\cos ({\omega }_{0}t-{\phi }_{\tau })+{\phi }_0(t))), \end{gathered}$$(5)where $m$ represents the coefficient of AOIM; ${\omega }_0$ is the modulation frequency; $A$ is the DC component of the interference signal and $B$ is the AC component of the interference signal; $C$ is the depth of phase modulation; ${\phi }_{\tau }={\omega }_0\tau$ represents CPD due to the delay in the optical path; ${\phi }_m$ is the phase difference between AOIM and the laser center frequency modulation.

Upon analog-to-digital conversion, the quantization error of the detected interference signal emerges. By increasing the ADC bits to enhance the acquisition accuracy, the error can be minimized, so the effect of quantization error is ignored in this paper. Finally, during PGC demodulation, the interference signal is split into two paths. In practice, higher-order Bessel function expansions and mB related terms are ignored to achieve simplification due to lower impact. After the first and second harmonic multiplication and filtering, the output results, $u_1$ and $u_2$, are expressed as $$\begin{gathered} u_1=-B{J}_1(C\cos {\phi }_{\tau })J_0(C\sin {\phi }_{\tau })·(1+J_2(C\sin {\phi }_{\tau })/J_0(C\sin {\phi }_{\tau }))·\sin ({\phi }_0(t)) \\ +\frac{1}{2}mA\cos \mathrm{\Delta }{\phi }_{m\tau }+g({\omega }_0), \end{gathered}$$(6)$$\begin{gathered} u_2=-B{J}_2(C\cos {\phi }_{\tau })J_0(C\sin {\phi }_{\tau })·\cos ({\phi }_0(t)) \\ +B{J}_2(C\sin {\phi }_{\tau })J_0(C\cos {\phi }_{\tau })·\cos ({\phi }_0(t))+h({\omega }_0), \end{gathered}$$(7)where $J_0,J_1$, and $J_2$ are zeroth-order, first-order, and second-order Bessel functions, respectively; $\mathrm{\Delta }{\phi }_{m\tau }={\phi }_m-{\phi }_{\tau }$; $g({\omega }_0)$ is the residual error after the first harmonic filter; $h({\omega }_0)$ represents the residual error after the second harmonic filter.

Therefore, as illustrated in Fig. 4, the FMI demodulated by the PGC algorithm can be regarded as consisting of two data acquisition systems with multiplicative and additive error terms and an Arctan algorithm. After ${\phi }_0(t)$ passes through ${\mathrm{ADC}}_1^{*}$ and ${\mathrm{ADC}}_2^{*}$, the calculated phase ${\phi }^{\prime }(t)$ with errors can be determined by division and Arctan algorithms. The phase output result of the solution system is expressed as follows: $${\phi }^{\prime }(t)=\arctan \left(\frac{M_{m,\tau }·M_{\tau 1}·\sin ({\phi }_0(t))+A_{m,\tau }+g({\omega }_0)}{M_{\tau 2}·\cos ({\phi }_0(t))+A_{\tau 2}+h({\omega }_0)}\right),$$(8)where $M_{m,\tau }$ and $A_{m,\tau }$ are errors introduced by AOIM and CPD, which are multiplicative error and additive error, respectively. $M_{\tau 1}$ and $M_{\tau 2}$ are the multiplicative errors introduced by CPD. $A_{\tau 2}$ is the additive error introduced by CPD into the cosine signal.

Therefore, the above errors can be briefly recorded as $$\{\begin{array}{l}{M}_{m,\tau }=1+J_2(C\sin {\phi }_{\tau })/J_0(C\sin {\phi }_{\tau }),\\ M_{\tau 1}=-B{J}_1(C\cos {\phi }_{\tau })J_0(C\sin {\phi }_{\tau }),\\ M_{\tau 2}=-B{J}_2(C\cos {\phi }_{\tau })J_0(C\sin {\phi }_{\tau }),\\ A_{m,\tau }=\frac{1}{2}mA\cos \mathrm{\Delta }{\phi }_{m\tau },\\ A_{\tau 2}=B{J}_2(C\sin {\phi }_{\tau })J_0(C\cos {\phi }_{\tau })·\cos ({\phi }_0(t)).\end{array}$$(9)

Analyzing the influence of each error term on the measurement results. (1)For terms $M_{\tau 1}$ and $M_{\tau 2},|M_{\tau 1}|=|M_{\tau 2}|$ can be obtained when the value of $C\cos {\phi }_{\tau }$ equals first- and second-order Bessel functions. Therefore, ideal solution results can be obtained by choosing a reasonable value of phase modulation depth $C$, while ignoring other error terms.(2)However, there are some issues with the approach mentioned in (1). First, the same time delay $\tau$ will result in different carrier phase delays ${\phi }_{\tau }$ at various modulation frequencies ${\omega }_0$, so different $C$ values need to be adjusted accordingly. Second, the error introduced by optical path delay only considers the multiplicative errors in (1), but additive error $A_{\tau 2}$ cannot be avoided. Therefore, it is necessary to implement compensation measures to adapt ${\omega }_0\tau$. For instance, in the off-line compensation method, the average delay can be determined through several experiments, and an appropriate circuit delay can be introduced according to the value of ${\omega }_0$, ensuring ${\phi }_{\tau }=k\pi$, $k\in Z$. In the case of the online compensation method, delay effects can be eliminated by introducing an online compensation unit.(3)For error terms $M_{m,\tau }$ and $A_{m,\tau }$, we can think of compensation methods in terms of $\mathrm{\Delta }{\phi }_{m\tau }$ or $m$. First, we can benefit from the phase compensation methods in (2), so that $\mathrm{\Delta }{\phi }_{m\tau }={\phi }_m-{\phi }_{\tau }=(2k+1)\pi /2$, $k\in Z$, in order to eliminate the influence of $m\cos \mathrm{\Delta }{\phi }_{m\tau }$. Second, the effective approach is to design a light-intensity compensation system or establish a reference optical path to reduce the impact of the AOIM coefficient $m$ on the solution system.(4)Residual errors, $g({\omega }_0)$ and $h({\omega }_0)$, caused by the filters are due to the non-ideal nature of the LPF in the PGC algorithm. These errors result from spectrum aliasing and other factors. Hence, it is essential to design the attenuation coefficient and cutoff frequency of LPF reasonably to mitigate the influence of residual errors.

After eliminating the errors caused by AOIM and PMD through existing techniques [16–20], this section then resolves the errors introduced by CPD in Eq. (5) under high-speed and long-distance measurement conditions. For the FMI, the essence of the motion of the object being measured is the Doppler shift. Therefore, the motion frequency of the object $f_o$ can be equivalent to the motion speed $v_o$, as $f_o=2v_o/\lambda$. The presence of a motion frequency induces a shift in the interference signal spectrum. Hence, when employing a certain modulation frequency, the spectrum of the interference signal shifts with the increasing motion speed of the measured object. If the speed continues to rise, spectrum aliasing occurs, ultimately rendering the interference demodulation system dysfunctional. It can be stated that the modulation frequency influences the theoretical maximum measurement speed that the interferometric system can demodulate. Considering a measurement speed index in this paper of 1.5 m/s, corresponding to a motion frequency of 2 MHz, the frequency of the modulating signal was finally selected as 10 MHz.

The impact of CPD on the equivalent acquisition bits and SINAD is examined using open-source numerical simulation software. As shown in Fig. 5, assuming that the object is in sinusoidal motion, Eq. (5) serves as the interference input signal, neglecting all other errors except for CPD, and the equivalent acquisition bits and SINAD are computed for various CPDs, ranging from 0 to $2\pi$. It can be observed that CPD exerts a substantial influence on the equivalent acquisition bits and SINAD of the first and second harmonics; particularly, when approaching odd multiples of $\pi /2$ or $\pi /4$, the equivalent acquisition bits and SINAD for the interference signal drop sharply, resulting in the inability to resolve the signal. Based on the previous analysis, increasing the carrier modulation frequency is essential to achieve high-velocity measurements, but it is inevitable that the CPD will increase significantly under the same system parameters. To delve further into this, certain assumptions are made. The modulation frequency of the carrier signal is set at 10 MHz, corresponding to a period of 100 ns. Consequently, the CPD of the light wave reaches 90 deg after a 1/4 period, which equals 25 ns. As a result, when conducting large-range measurements and measuring objects at varying distances, the corresponding CPD of the system as well as SINAD will be different.

The following simulation experiments were conducted to further elaborate on the requirements of large-range and high-velocity measurements. Figures 6(a) and 6(b) correspond to the variation of the equivalent acquisition bits and SINAD for the first and second harmonics with the work distance when the modulation frequency is 3 MHz, 7 MHz, and 10 MHz. Figures 6(c) and 6(d) correspond to the variation of the equivalent acquisition bits and SINAD for the first and second harmonics with the work distance when the initial delay angle is 0 deg, 45 deg, 90 deg, and 135 deg.

As shown in Figs. 6(a) and 6(b), it can be seen that for the requirement of low-speed measurement (corresponding to low modulation frequency), the equivalent acquisition bits and SINAD are not greatly affected by the variation of the work distance within the range of 2 m. However, as the measurement speed increases (the corresponding modulation frequency also increases), for example, when the modulation frequency is 10 MHz, the equivalent acquisition bits and SINAD of the second harmonics have huge errors at the work distance of about 1.9 m. The above analysis of the measurement speed shows that when the initial delay angle is zero and under the same measurement conditions, the increase in the measurement speed also increases the CPD, which has a drastic effect. Then the effects of different initial delay angles when the modulation frequency is fixed will be studied below. As shown in Figs. 6(c) and 6(d), the modulation frequency is 10 MHz at this time. Compared with the curve when the initial delay angle is zero, it can be seen that when the initial phase keeps increasing, the error caused by the initial phase angle is directly superimposed on the error caused by the modulation frequency. In addition, the errors brought by different initial delay angles to the system are also different, and the initial delay angles in the actual experiments are also uncertain, which makes the errors caused by the initial delay angles difficult to avoid. Therefore, real-time dynamic adjustments of CPD are essential to ensure the accuracy and correctness of the demodulation results.

Building on the previous analysis, there exists a carrier phase delay ${\phi }_{\tau }$ between the interference signal and the internal reference signal, due to the time delay of $\tau ={\tau }_d+{\tau }_s$ after the interference signal passes through the detection optical path, where ${\tau }_d$ represents the time spent in the work distance, and ${\tau }_d$ is variable due to the object’s motion. On the other hand, ${\tau }_s$ is the time taken by the other remaining optical paths (except the work distance), so it is a constant. During large-range measurements, the CPD progressively increases with the movement of the object, causing a decrease in the signal’s SINAD after mixing, and it may even lead to the failure of the demodulation algorithm. Therefore, it is necessary to design a CPD compensation algorithm, which is divided into three stages.

First stage: when the object being measured has not started moving, and the system has just been powered on, a fixed CPD caused by circuit delay is generated. At this point, the CPD compensation amount of the reference signal generation module is adjusted up or down at fixed intervals. Simultaneously, the amplitude of the sinusoidal signal in the two orthogonal signals is monitored to determine whether it has reached its maximum. If the maximum is reached, the CPD compensation amount ${\phi }_s$ is locked at this time to complete the static CPD compensation.

Second stage: after the completion of static fixed CPD compensation, the measured object begins to move, and the CPD gradually increases with the motion of the object, resulting in a dynamic CPD. During this stage, the change in dynamic carrier phase delay ${\phi }_d$ can be calculated based on the phase information already demodulated by the PGC-Arctan algorithm. The dynamic carrier phase delay ${\phi }_d$ can be expressed as $$\begin{gathered} {\phi }_d={\omega }_0{\tau }_d=2\pi f_0·\frac{2L}{c} \\ =\frac{f_0}{v}{\phi }^{\prime }(t), \end{gathered}$$(10)where $f_0$ is the laser modulation frequency; $c$ is the speed of light in a vacuum; $v$ is the output frequency of laser.

Third stage: the static CPD compensation ${\phi }_s$ from the first stage is added to ${\phi }_d$, resulting in ${\phi }_{\tau }={\phi }_d+{\phi }_s$, which is then fed back to the CPD compensation input of the reference signal generation module to achieve real-time dynamic CPD compensation.

The experimental process depicted in Fig. 7 unfolds as follows. The DFB laser is current-modulated by the sinusoidal signal generated by the direct digital synthesis (DDS) unit and the digital-to-analog converter (DAC). The output light is transmitted through a single-mode fiber (SMF) and passes through the fiber isolator and the optical fiber circulator (OC). Then it travels through the GRIN lens and interferes, and then returns to the OC. The resulting interference signal enters the PGC demodulation system after passing through the ADC. At this point, both the static CPD caused by the circuit delay and the dynamic CPD arising from the motion of the object have been calculated and stored in the compensation algorithm. Following phase compensation, the first and second harmonic carrier signals are multiplied by the interference signals, followed by low-pass filtering. After dividing the two orthogonal signals, the inverse tangent operation is applied to obtain the phase to be measured, thus achieving phase demodulation.

In the experiment, a DFB laser serves as the dedicated laser source for the FMI, and it is driven by the DLC laser driver (DLC PRO, Toptica, Germany). The stable interference signal is detected by a photodetector and converted into a digital signal using a 16-bit ADC. The signal processing board is used for signal processing, as well as to generate sinusoidal modulated signals, which are output by a 14-bit DAC. The field programmable gate array (FPGA) is used to implement the PGC optimization algorithm, and the final demodulation results are subsequently sent to the personal computer via the universal serial bus (USB).

To meet the requirements of equivalent signal acquisition accuracy for the phase-generated carrier demodulation lock-in amplifier unit mentioned earlier, a multi-functional programmable signal generator is set up. In order to measure the static measurement limit of the signal processing unit of the FMI, several steps were taken. The first step was to set ${\phi }_0(t)$ as a fixed value in Eq. (5) to simulate the situation where the measured object was in a static state, and then connect the signal generated by the signal generator to the self-designed signal processing circuit board. The second step was to display the collected digital fluctuations of the first and second harmonics in Fig. 8. Figure 8(a) illustrates the test results of equivalent acquisition accuracy for the first harmonic lock-in amplifier, with the peak-to-peak value of equivalent digital quantity fluctuation being 24. Figure 8(b) shows the test result of equivalent acquisition accuracy for the second harmonic lock-in amplifier, with the peak-to-peak value of equivalent digital quantity fluctuation being 25. According to the relation between digital fluctuation and equivalent acquisition bits, the equivalent acquisition accuracies for the first and second harmonic lock-in amplifiers are 11.42 bits and 11.36 bits, respectively. These values are close to each other and align with the assumption made in Section 2 regarding signal processing accuracy analyses. Furthermore, both values are better than 10.30 bits, proving that the unit design can meet the requirements of 0.1 nm displacement measuring resolution.

The experiments conducted above confirm that the phase-generated carrier demodulation unit meets the requirements of subnanometer resolution design. The third step was to test the accuracy of the carrier demodulation generated by the phase-generated carrier lock-in amplifiers combined with the arctan unit, as shown in Fig. 9. The signal demodulation output of the FMI is collected over 3 h, completing the long-term static phase test of the system. According to the test results, the peak-to-peak value fluctuation of the digital quantity is eight, and the corresponding displacement and jitter amount is less than 0.1 nm.

On the basis of previous analyses, it is evident that CPD changes two orthogonal signals with equal amplitude into orthogonal signals with different amplitudes, thereby directly impacting the demodulation of SINAD. In the context of a large range and high measurement velocity, the CPD rises with the increasing displacement of the object being measured. The CPD compensation module can be verified by comparing the two amplitude ratios of the two orthogonal signals before and after compensation. The following specific steps were taken. First, a programmable signal generator was employed to generate an analog interference signal with the object to be measured in a constant motion state and with a dynamic CPD change. The signal was then transmitted to the demodulation system, and the data of the two pairs of orthogonal signals before and after CPD compensation were collected. As demonstrated in Fig. 10, a comparison was made between the results measured by the proposed method and the methods described in previous research [24,25] through experiments.

Figure 10(a) explores the relationship between the calculated carrier phase delay ${\phi }_{\tau }^{\prime }$ and the phase to be measured ${\phi }_0$. Analysis reveals that the blue curve, representing the former research method [24], exhibits spikes at certain specific points. This occurs because when ${\phi }_0$ is $k\pi$, $k\in Z$, the value obtained by dividing the sine and cosine signals is either zero or infinite, so an accurate carrier phase delay value cannot be obtained. Conversely, by observing the red curve representing the proposed method, it can be observed that ${\phi }_{{\tau }^{\prime }}$ remains unaffected by the change in ${\phi }_0$. Second, the ${\phi }_{{\tau }^{\prime }}$ was compared to the true value of the carrier phase delay, ${\phi }_{\tau }$ [Fig. 10(b)]. It was found that when ${\phi }_{\tau }$ is $(2k+1)\pi /4$, $k\in Z$, the blue curve, representing the method of former research [25], also exhibits spikes. This happens because the presence of CPD introduces a multiplicative error term $\cos (2{\phi }_{\tau })$ in the cosine signal, leading to the denominator term in the division signal under these specific points becoming zero. Consequently, an accurate carrier phase delay value cannot be obtained. In contrast, the red curve, representing the proposed method, indicates that ${\phi }_{{\tau }^{\prime }}$ is consistent with ${\phi }_{\tau }$ and remains unaffected by ${\phi }_{\tau }$.

In summary, when ${\phi }_0$ and ${\phi }_{\tau }$ take certain specific points, the former research methods yield error-prone results with significant errors, resulting in inaccurate CPD compensation values. In contrast, the results calculated by the proposed method are accurate and unaffected by these specific points.

Resolution measurement experiments were conducted using a programmable signal generator to generate an analog interference signal with the step motion state of the object to be measured. Various bit step values, 0.36 nm, 0.20 nm, 0.15 nm, and 0.10 nm, were set. The results obtained are displayed in Fig. 11.

It can be observed that the steps are clear and distinguishable, indicating that the electrical test resolution of the measuring system is 0.10 nm.

As shown in Fig. 12, experiments were conducted to measure the speed of the system. A programmable signal generator was used to generate an interference signal simulating the uniform motion state of the measured object at speeds of 383.2 mm/s, 766.4 mm/s, 1149.6 mm/s, and 1532.8 mm/s. The collected data after demodulation were uploaded to the computer for the least squares fitting. A total of 4096 collected points were selected for data fitting, and the fitting results under different speeds were obtained.

Figure 13 illustrates the results of displacement residuals at speeds of 383.2 mm/s, 766.4 mm/s, 1149.6 mm/s, and 1532.8 mm/s. As observed from the figure, the residual increases with the increasing object’s motion speed. This is attributed to the increase in the motion frequency of the object to be measured, which broadens the spectrum of the interference signal, ultimately reducing the stopband attenuation of the low-pass filter.

The displacement standard deviation was calculated based on the fitting results and the residual distribution. The relationship between the standard deviation of measurement and the frequency of motion is plotted in Fig. 14.

It reveals that when the equivalent frequency of the motion of the object to be measured is within the range of 2000 kHz, that is, when the motion speed of the object to be measured is within 1.5 m/s, the measurement standard deviations are consistently below 0.5 nm. This shows that the demodulation system has achieved subnanometer precision at a high-speed measurement of 1.5 m/s.

The new generation of ultra-precise FMI plays a vital role in displacement measurement with subnanometer resolution, thanks to its advantages of compact measurement space and high integration. Phase-generated carrier demodulation technology based on internal modulation is a commonly used phase demodulation method. However, this algorithm is only suitable for small distance and low-speed measurement conditions, and nonlinear errors are introduced under high-speed and large-range modulation.

This paper proposes, for the first time, a resolution equivalent model, which equivalently models the modulation and PGC demodulation processes as a signal acquisition system, enabling the total error generated by all internal links to be characterized using $N^{*}$ in the resolution equivalent system, providing a basis for error allocation in subnanometer resolution design. Additionally, this paper offers a theoretical analysis of nonlinear errors and error sources generated during high-speed measurement. It presents an adaptive compensation algorithm combining both dynamic and static CPD to address the errors introduced by CPD during PGC demodulation. Experimental results demonstrate that when the measurement speed is 1.5 m/s, the measurement error caused by CPD can be reduced to less than 0.5 nm, thus achieving subnanometer resolution accuracy requirements.

In the future, we aim to further analyze other error sources presented here and provide practical solutions to enhance measurement accuracy.