The high $Q$ whispering gallery mode (WGM) resonator is one of the ideal platforms for high-precision sensing. From the physical level, its main sensing mechanism is based on the change of the equivalent cavity length of the resonator caused by the physical parameter to be measured. The change of the resonator conditions causes the center frequency of the resonator mode to shift, and demodulation of the frequency shift or the phase change is carried out [1–7]. Thus, high $Q$ optical modes corresponding to resonant spectral linewidths on the order of MHz or even kHz can provide better detection limit. This type of sensing, known as the dispersive coupling response, has been shown to be highly competitive in detecting a range of physical quantities with high precision. Examples of such sensing techniques include acoustic sensing [8–11], magnetic sensing [12–15], and inertial sensing [16–21].

Nevertheless, there is a dearth of convincing demonstration applications in quasi-static sensing measurements. This is due to the susceptibility of high $Q$ optical resonators to low frequency noise such as temperature. In the case of a silica resonator, for example, a temperature change of 0.001 K results in a resonator frequency shift on the order of MHz, which is clearly far more than the frequency change due to the weak signal to be measured. The Pound–Drever–Hall frequency locking technique is capable of mitigating the impact of low frequency noise, such as temperature fluctuations, through real-time error signal feedback [22,23]. However, this approach also presents a challenge in accurately extracting the true low frequency information to be measured. Other methods, such as the construction of sensing matrices or the implementation of multimode inter-correlation algorithms for temperature decoupling [24–28], frequently encounter constraints in terms of the detection limit or response time [29].

A dissipative coupling response mechanism based on a resonator is a more competitive solution for quasi-static, high-precision measurement requirement. The self-referenced detection method is inherently capable of suppressing low frequency common-mode noise, such as temperature, resulting in a lower theoretical detection limit [1,30–32]. Xiao et al. from Peking University and our previous study showed that the loading of the signal to be measured with a certain frequency leads to a periodic change in the decay rate of the evanescent field outside the resonator [33,34]. If enough oscillating signals can be superimposed on the Lorentz line pattern of the resonant mode, the high-precision real-time demodulation that can be carried out can be used to solve the fast response problem faced by the dissipative coupling mechanism. However, these methods are limited to signals to be measured with a certain frequency and cannot yet constitute an effective means of sensing quasi-static physical quantities. This is due to the fact that the lower frequency of the quasi-static signal cannot be effectively superimposed on the transmission spectrum to produce a response.

Therefore, we propose a ${\text{LiNbO}}_3$ resonator self-modulation method within the framework of the dissipative coupling response for high-precision sensing measurements of quasi-static physical quantities. The active modulation of the signal to be measured by the ${\text{LiNbO}}_3$ resonator solves the problem of real-time demodulation of quasi-static signals and avoids the losses and complexity introduced by additional modulation devices. The variation in the amplitude of the filtered signals can be used to directly measure the displacement variation without the need for complex data processing. Furthermore, the low frequency region with elevated $1/f$ noise is upconverted to the high frequency low noise region, thereby enhancing the detection limit. The results of the finite element simulation demonstrate that the structural characteristics of the resonator have a significant impact on the evanescent field attenuation rate. Consequently, optimization of orthogonal polarization by dissipation engineering can significantly improve the sensing response by more than a factor of two. Experimentally, we used quasi-static displacement sensing as a demonstration. A customized $Q=2.09\times {10}^7$${\text{LiNbO}}_3$ resonator is used as the high frequency modulation and sensing element, and a movable WGM resonator in the evanescent field is used as the displacement loading unit. The displacement sensitivity is 0.0416 V/nm, and the Allan deviation $\sigma$ indicates a sensing system bias instability of 0.205 nm, which is the best result we know for resonator displacement sensing in the quasi-static range. The proposed scheme demonstrates competitiveness in quasi-static high-precision sensing and provides solutions for the high-precision real-time detection of low or very low frequency acceleration, pressure, nanoparticles, or viruses.

In this paper, a dissipative coupling response mechanism is used to read the broadening or narrowing of the transmission spectrum due to displacement changes. Since both the initial and the transmission spectra affected by the displacement signal experience the same noise, most of the common mode noise (e.g., temperature noise) can be eliminated using this self-referencing differential detection technique. On the other hand, high-sensitivity detection of low frequency signals (typically in the Hz to kHz range) is limited by the prevalence of $1/f$ noise in electrical and optical systems [35]. The transmission spectrum has been successfully modulated using the Pockels coefficient of ${\text{LiNbO}}_3$ crystal. The low frequency displacement signal has been upconverted to the modulation frequency (50 kHz), resulting in a reduction of the noise floor by two orders of magnitude. At the same time, the weak changes in the transmission spectral linewidth are converted into the increase or decrease of the carrier signal envelope amplitude by the resonance enhancement, and the sub-nanometer scale (0.205 nm) displacement detection precision is obtained.

The optimization of the sensing mechanism and dissipation engineering based on the radiation loss of the resonator is shown in Fig. 1. A schematic structure of the prism coupling of a self-modulated ${\text{LiNbO}}_3$ WGM resonator is shown in Fig. 1(a). The detailed displacement sensing principle is shown in Fig. 1(b). The initial transmission spectrum is a smooth Lorentz line shape, and the spectrum oscillates after the ${\text{LiNbO}}_3$ resonator is modulated by a periodic electric field. The normalized response amplitude superimposed onto the transmission spectrum is read by bandpass filtering, while the response amplitude of the signal is amplified using the transmission spectrum resonance enhancement. When the coupling distance $d$ between the resonator and the prism changes with displacement, the coupling loss $\kappa$ of the transmission spectrum also changes. Specifically, the transmission spectrum is broadened or narrowed, while the amplitude of the filtered signal changes, and the amplitude after the displacement change is subtracted from the initial filtered amplitude to obtain the change due to the displacement change ($\mathrm{\Delta }V=V_1-V_2$). Therefore, the variation of the displacement can be inverted by reading the amplitude variation of the filtered signal.

Due to the anisotropy of ${\text{LiNbO}}_3$ crystal, the effective refractive indices of the different mode resonances in the Z-cut ${\text{LiNbO}}_3$ resonator are significantly different for TM ($n_o=2.21$) and TE ($n_e=2.14$) modes. Furthermore, the finite element simulation analysis reveals that the optical resonance modes are better confined inside the resonator due to the higher effective refractive index of the TM modes, while the evanescent field tail decays faster outside the resonator. The evanescent field decay outside the resonator can be approximated in the form of an exponential decay: $E\propto \exp [-2\pi d\surd (n_{\mathrm{eff}}^2-n_0^2)/\lambda ]$, where $d$ is the horizontal distance outside the resonator, $n_{\mathrm{eff}}$ is the effective refractive index of the resonator, $n_0$ is the refractive index of the air, and $\lambda$ is the laser wavelength. This allows the TM mode to propagate with less loss at the resonator-air interface with a high refractive index difference, with a stronger binding force. As demonstrated in Figs. 1(c) and 1(d), the decay rate of the evanescent field is enhanced with decreasing coupling distance and decreasing resonator diameter, and the radius of curvature of the resonator sidewalls has almost no effect (Appendix A). In the same case, the rate of evanescent field attenuation of TM mode is about two times that of TE mode, so the sensitivity of the displacement response can be effectively enhanced by choosing TM mode combined with dissipative engineering optimization in this sensing system. On the other hand, modulation of the resonator using the Pockels effect of ${\text{LiNbO}}_3$ material can achieve frequency upconversion and reduce the influence of $1/f$ noise, while solving the difficulty of the fast response of low frequency signals. Based on the above principles, we provide a new solution for high-precision sensing using a prism coupling platform with electro-optic modulation of the ${\text{LiNbO}}_3$ resonator.

At the resonant frequency, the electromagnetic field $a$ in the WGM resonator can be expressed by the following equation: $$\frac{\mathrm{d}a}{\mathrm{d}t}=-(\frac{{\kappa }_i}{2}+\frac{{\kappa }_e}{2}+\frac{{\kappa }_s}{2}+i\mathrm{\Delta }{\omega }_L)a+\sqrt{{\kappa }_e}{a}_{\mathrm{in}},$$(1)where $\mathrm{\Delta }{\omega }_L={\omega }_L-{\omega }_0$ denotes the detuning of the laser to the resonant frequency of the resonator, and ${\omega }_L$ and ${\omega }_0$ denote the intrinsic resonant frequencies of the laser and the WGM resonator, respectively. ${\kappa }_i$ is a constant value, representing the intrinsic loss in the resonator, ${\kappa }_e$ is the coupling loss due to the prism, and ${\kappa }_s$ is the scattering loss in the prism coupling region. The external loss is defined as ${\kappa }_{\mathrm{ex}}={\kappa }_e+{\kappa }_s$, and ${\kappa }_{\mathrm{ex}}$ is closely related to the coupling distance between the resonator and the prism. Here $a_{\mathrm{in}}$ denotes the normalized input amplitude, the signal amplitude received from the prism output is $a_{\mathrm{out}}=a\surd {\kappa }_e-a_{\mathrm{in}}$, and the normalized transmission through the prism is $T={|a_{\mathrm{out}}/a_{\mathrm{in}}|}^2$.

The ${\text{LiNbO}}_3$ material exhibits an excellent Pockels effect, whereby the refractive index is subject to modulation as a result of the application of an electric field. Consequently, the refractive index and intrinsic resonance frequency of the ${\text{LiNbO}}_3$ resonator are also subjected to periodic changes following the introduction of periodic modulation. The external nonlinear electric field modulation parameters are introduced into the electromagnetic field equations in the resonator: $$\frac{\mathrm{d}{a}_{\mathrm{\Omega }}}{\mathrm{d}t}=-[\frac{{\kappa }_i}{2}+\frac{{\kappa }_{\mathrm{ex}}}{2}+i(\mathrm{\Delta }{\omega }_L+\mathrm{\Delta }{\omega }_E)]a_{\mathrm{\Omega }}+\sqrt{{\kappa }_e}{a}_{\mathrm{in}},$$(2)where $\mathrm{\Delta }{\omega }_E$ denotes the resonance detuning resulting from modulation by an applied electric field, and the details of the derivation of the formulas are given in Appendix B.

In a resonator coupling system, the coupling distance perturbs the distribution of the mode field in the resonator by modulating the dissipative response (${\kappa }_{\mathrm{ex}}$). As the coupling distance increases, the coupling gap between the resonator and the prism increases and ${\kappa }_{\mathrm{ex}}$ decreases, at which point the response amplitude generated by the periodic modulation decreases. The normalized displacement sensitivity of the coupled system was analyzed theoretically as $S=\partial T/\partial d=0.00562{\mathrm{nm}}^{-1}$, and the detailed derivation process is described in Appendix C. In the experiment, the laser injection power is $P_{\mathrm{in}}=800\mathrm{\mu W}$, and the conversion gain coefficient $G=1\times {10}^4\mathrm{V}/\mathrm{W}$. Therefore, the theoretical displacement sensitivity of the coupled sensing system can be calculated as $S_d=\mathrm{\Delta }V/\mathrm{\Delta }d=S\times P_{\mathrm{in}}\times G=0.045\mathrm{V}/\mathrm{nm}$.

Based on the above theoretical analysis, we prepared ${\text{LiNbO}}_3$ crystal resonators with different radii (e.g., $R=1.5,2.5,4,5\mathrm{mm}$), keeping the same radius of curvature of the sidewalls as much as possible, close to the columnar surface, and all with a thickness of 0.1 mm. The enhancement of the evanescent field by the dissipation engineering is experimentally verified. The resonance modes are primarily adjusted by means of controlling the polarization state and angle of incidence of the laser. Figure 2 shows the test results for several different resonators. As illustrated in Figs. 2(a)–2(d), the trend of external loss (${\kappa }_{\mathrm{ex}}$) with coupling distance ($d$) is demonstrated for TE and TM modes in various diameter resonators. Fitting the data reveals an exponential decay of the external loss as a function of $d$. In the sensing system, the primary concern pertains to the rate of evanescent field attenuation, otherwise referred to as the rate of loss variation with distance ($\partial {\kappa }_{\mathrm{ex}}/\partial d$). Figures 2(e) and 2(f) demonstrate the evanescent field attenuation rate of the resonator in relation to the coupling distance for both TM and TE modes. It is found that (1) the smaller the radius of the resonator at a fixed coupling distance, the faster the rate of evanescent field decay for the TM and TE modes. The larger the radius of the resonator, the closer the coupling prisms are to the resonant modes, which has less effect on the effective refractive index, and the evanescent field attenuation becomes less significant. (2) For the same case, the decay rate of TM mode is much larger than that of TE mode in the range of the resonator evanescent field. In order to obtain more sensitive readings of the displacement signals, we select TM resonant modes with a radius of 1.5 mm for sensing measurements in subsequent experiments.

The sub-nanometer displacement sensing system based on self-modulation of the ${\text{LiNbO}}_3$ resonator is shown in Fig. 3(a). Figure 3(b) shows the physical ${\text{LiNbO}}_3$ resonator with gold electrodes deposited on the top and bottom surfaces, and its transmission spectrum with a loading of $Q=2.09\times 10$⁷. Further we analyzed the response amplitude of the signals in the resonator for different modulation voltages and frequencies; experimentally the highest response amplitude corresponds to the critically coupled state at a modulation frequency of 50 kHz and a voltage of 0.3 V. The original unit of the response amplitude is the voltage, which has been normalized for improved comparison in Fig. 3(c). The response of the modulation frequency and voltage to the resonator with varying $Q$-values was also calculated. It was found that the response amplitude was highest at $Q=1.5\times {10}^7$, corresponding to a higher sensing sensitivity, at a laser tuning rate of 12,000 MHz/s. We designed the ${\text{LiNbO}}_3$ resonator with a $Q$ of $2.09\times {10}^7$, which is close to the $Q$ value corresponding to the highest response and provides highly sensitive displacement sensing.

In this system, a tunable narrow linewidth laser (TOPTICA CTL 1550) is used as the light source. The polarization controller is responsible for regulating the polarization state of the input light. Coupling of evanescent waves is achieved using a high refractive index rutile prism coupled to a ${\text{LiNbO}}_3$ resonator based on phase matching theory. The signal generator provides a sinusoidal signal, which is used to periodically modulate the ${\text{LiNbO}}_3$ resonator for the purpose of frequency upconversion. Optical signal passing through the resonator is output to a photodetector (PDB435C), where it is converted into an electrical signal. The original modulated transmission spectrum is filtered using a bandpass filter, and the filtered signal is finally displayed on an oscilloscope (Tektronix MSO64). The change in coupling distance can be deduced from the change in amplitude of the filtered signal.

Further, the displacement sensing system was tested, where Fig. 4(a) shows the transmission versus coupling distance for the coupled system with different losses (${\kappa }_e/{\kappa }_{\mathrm{ex}}$), and the larger the coupling loss ${\kappa }_e$ in the external loss ${\kappa }_{\mathrm{ex}}$, the closer the system is to the coupling ideal case.

In our system, the scattering loss ${\kappa }_s$ brought by the coupling prism leads to the non-ideal coupling, and the experimental values (red dots) are fitted with ${\kappa }_e/{\kappa }_{\mathrm{ex}}=1/2$, i.e., ${\kappa }_e={\kappa }_s={\kappa }_{\mathrm{ex}}/2$, in the displacement sensing system.

The advantages of electro-optic modulation-induced frequency upconversion in terms of noise rejection are shown in Fig. 4(b). The black line is the lowest and shows the electrical noise level of the photodetector itself; the gray line shows the noise of the laser; the red line is the noise of the transmitted spectral line after the resonator has been coupled to the prism; and the blue line is the noise background of the resonator subjected to periodic electro-optic modulation. It is evident that in the region below kHz, the system noise is elevated by the influence of $1/f$ noise. By upconverting the low frequency weak displacement signal to 50 kHz through the use of ${\text{LiNbO}}_3$ resonator self-modulation technology, the noise floor is reduced by two orders of magnitude in comparison to the low frequency region, thereby significantly enhancing the detection limit of the sensing system. In the absence of supplementary modulators to provide upconversion modulation frequency within the system, the ${\text{LiNbO}}_3$ resonator is used directly as an electro-optic modulation device. This approach enables highly efficient electro-optic modulation of the light field localized modes, thereby markedly enhancing the sensing response sensitivity of the system through resonance enhancement. On the other hand, fast modulation paves the way for real-time reading of quasi-static signals while saving space in the sensing system and avoiding the losses and complexity introduced by additional modulation devices. This provides novel solution ideas for weak signal detection in restricted space.

In order to test the displacement sensitivity of the sensing system, the coupling distance was reduced from 300 to 50 nm with a step size of 10 nm, and each reduction was held for approximately 2.5 s. In order to facilitate a more evident comparison of the step signals, Fig. 4(c) has been prepared, in which the step signals with a displacement gap of 20 nm have been selected for demonstration purposes. As illustrated in Fig. 4(d), the response amplitude is contingent on the coupling distance, exhibiting a system sensitivity of $S=|\mathrm{\Delta }V/\mathrm{\Delta }d|=0.0416\mathrm{V}/\mathrm{nm}$ within the elevated response range of 50–100 nm. The linear fit is characterized by Pearson’s $r=-0.95$ and $R^2=90.2\%$.

The Allan deviation of the displacement sensing system based on ${\text{LiNbO}}_3$ resonator self-modulation was analyzed through the monitoring of the output signal of the sensing system at rest over an extended period of time. The Allan deviation $\sigma$ is used for the characterization of the stability of the sensor, thereby enabling the detection limit of the system to be resolved by describing the noise and stability as a function of the sampling time $\tau$ [36,37]. It is a time domain metric that can be calculated from the analysis of long samples of data obtained from stationary sensor outputs. Using the same calculation method, the noise coefficients of each noise term are extracted by fitting and combining the slopes of the lines defined by $k$. Figure 5 illustrates the long-term stability of the sensing system and the Allan deviation analysis result.

The random walk noise (equivalently, noise density), which decreases with the square root of time, is equal to $q{\tau }^{-0.5}$, where $\tau =1\mathrm{s}$. Furthermore, the region where the slope of the Allan deviation with respect to the average time is zero is characterized as bias instability noise. Therefore, the displacement bias instability is equal to $q{\tau }^0$. The random walk noise of the system is $0.157\mathrm{nm}/\surd \mathrm{Hz}$, and the displacement bias instability is 0.205 nm. The test results of this system are more advanced compared to other resonator-based displacement sensing systems. Among them, the random walk noise is mainly related to the temperature variation, which can be reduced by improving the stability of temperature control. The bias instability indicates the minimum of readings limited by scintillation noise and initial coupling distance $d_0$ or optical field fluctuations.

The long-term stability of the system is of critical importance for the application of displacement sensing systems. Here the bias instability can represent the detection limit of the displacement sensing system. As demonstrated in Fig. 6, a comparison is made between the detection limits of optical resonator-based displacement sensing [38–44]. We designed a quasi-static displacement sensing system based on the self-modulation of the ${\text{LiNbO}}_3$ resonator, and its detection limit is significantly better than in other studies.

In conclusion, we present a sub-nanometer displacement sensing method based on self-modulation of the ${\text{LiNbO}}_3$ resonator. The method enhances the detection limit by utilizing the amplitude of the modulation response enhanced by a high $Q$ resonator. Fast modulation achieves high-precision real-time reading of quasi-static weak signals. Self-referencing readout using dissipative sensing avoids the effect of common mode noise as much as possible. Direct modulation of the resonator achieves frequency upconversion and noise floor suppression by 2 orders of magnitude. The experimentally fabricated ${\text{LiNbO}}_3$ crystal resonator has a critical coupling load $Q$ of $2.09\times {10}^7$ and a coupling depth greater than 85%. It is of paramount importance that the sensing system exhibits good long-term stability in order to facilitate the highly sensitive detection of physical parameters. In this paper, weak displacement signals are detected using electro-optic modulation of a ${\text{LiNbO}}_3$ resonator, achieving displacement bias instability as low as 0.205 nm. The sensing system verifies precision displacement sensing measurements on the sub-nanometer scale. In the future, the integrated packaging of mass produced crystalline resonators can be used to provide core sensitive units for intelligent precision manufacturing, environmental vibration monitoring, and detection of biological cellular motions and molecular interactions. This method of using resonator direct modulation to achieve resonance enhancement of response amplitude as well as real-time reading of signals by frequency upconversion and dissipative response provides a new solution for ultra precision measurement of a wide range of physical quantities.