Atomic ensembles with large atom number $N_{\mathrm{at}}$ are desirable in precision measurements [1–3], since the measurement precision is proportional to $1/\sqrt{N_{\mathrm{at}}}$, the standard quantum limit determined by the atom projection noise (PN) [4,5]. Squeezed spin states (SSSs) [6,7] can be used to surpass this limit and improve sensitivities in atomic spectroscopy [4], interferometry [8], magnetometers [9], and atomic clocks [10]. However, experimental realization of large-scale SSSs is challenging mainly because of the more stringent requirements on the necessary signal-to-noise ratio, since the classical noise amplitude often scales as $N_{\mathrm{at}}$ and dominates over the atom PN proportional to $\sqrt{N_{\mathrm{at}}}$. The largest atom number for a spin-squeezed state was ${10}^8$ [11], and in cold atoms it was ${10}^6$ [7]. Only recently, we have escalated the atom number to ${10}^{11}$ for SSSs [9]. An unpolarized thermal state was “squeezed” with an atom number of about ${10}^{13}$ [12].

The major contributions to classical noise are associated with the imperfect state preparation of the coherent spin state (CSS) [6], the initial state for a normal spin squeezing experiment, as well as the unavoidable inhomogeneous background of electrical and magnetic fields. In general, to obtain spin squeezing, the CSS is prepared by optically pumping all the atoms to a particular hyperfine or Zeeman sublevel with resonant laser beams [13]. However, populations in other unwanted states are inevitable, even with sufficient laser power, due to various experiment imperfections. It is known that the adiabatic process can be useful when preparing a system to a desired eigenstate [14–16] where the state of the system follows the instantaneous eigenstate, but it has not been explored in the context of CSS preparation for SSSs.

Here, we develop and demonstrate the technique of adiabatic pulse control in the preparation of CSS for spin squeezing in a ${}^{87}\mathrm{Rb}$ ensemble of ${10}^{11}$ atoms contained in a macroscopic vapor cell. By adiabatically shutting down the optical pumping that is used to produce the CSS, one can eliminate the undesired spin component excited by the high-frequency Fourier components in the falling edge of pump beams. Such a spin component is referred to as “transverse spin” in the rest of the paper. To model the spin dynamics during the optical pumping, we consider a simplified $\mathrm{\Lambda }$-scheme, in which the excited state is coupled to two ground states of the Zeeman sublevels. Our experimental observations agree well qualitatively with the theory.

As reported in Ref. [9], we achieved spin squeezing by quantum nondemolition (QND) measurement via off-resonant atom–light Faraday interaction. The experiment setup [Fig. 1(a)] includes a four-layer magnetic shielding, containing a paraffin-coated $20\mathrm{mm}\times 7\mathrm{mm}\times 7\mathrm{mm}$ rectangular vapor cell and a set of coils for generating a homogeneous bias magnetic field of 0.71 G that gives a ground-state Zeeman splitting of about ${\mathrm{\Omega }}_L=2\pi \times 500\mathrm{kHz}$. The measured decay times for the ground state spin population and coherence are $T_1=125\mathrm{ms}$ and $T_2=20\mathrm{ms}$, respectively, with the latter mainly limited by residual magnetic field inhomogeneity. The $x$-polarized probe laser propagating along the $z$ axis is blue-detuned by 2.3 GHz from the $5S_{1/2},F=2\to 5P_{3/2},F^{\prime }=3$ transition of the D2 line. Its intensity is modulated at twice the Larmor frequency by an acousto-optic modulator to implement the stroboscopic quantum back-action evasion protocol [11], with an optimal duty cycle of 14%. The $S_y$ operator is measured by balanced polarimetry and lock-in detection in Ref. [9]. However, in this experiment, we also detect the signal associated with the residual ground state coherence by a spectrum analyzer, which has a flat frequency response and serves as a cross-check for the results from the lock-in amplifier.

First, atoms are prepared in the state $5S_{1/2}|F=2,m_F=-2⟩$ (with quantum number $m_F$ associated with the quantization axis along the $x$ axis, the direction of the magnetic field) by applying circularly polarized and spatially overlapped ${\sigma }^{-}$ pump and repump lasers propagating along $x$ [9,11]. We achieve up to 97.9% polarization of spin, as measured by the magneto-optical resonances [17]. The optimized laser powers are 50 mW for the repump laser and 5 mW for the pump laser, both with elongated Gaussian transverse intensity distributions. The probe laser mode has a symmetric Gaussian profile with $1/e^2$ beam diameter of 6 mm. All three fields cover nearly the entire cell volume.

To obtain quantum squeezing in a large ensemble, the greatest challenge is to overcome classical noises, which are often proportional to $N_{\mathrm{at}}$. Therefore, strict orthogonality between the polarized spin and the probe field’s wave vector is required to avoid the classical spin component in the y–z plane of the quantum noise measurement. Such alignment can be optimized using the intensity-modulated pump field as in a Bell–Bloom magnetometer configuration [18], which produces a large classical signal proportional to the mismatch between the pump’s wave vector direction and the bias magnetic field direction $x$. However, even after such fine-tuning, a small residual $\pi$ polarization component persists when viewed in the $x$-quantization basis, which, together with the ${\sigma }^{-}$ component, creates unwanted ground state coherence (associated with the superposition state $|F=2,m_F=-2⟩+ϵ|F=2,m_F=-1⟩$ where $ϵ\ll 1$) via a two-photon process, as shown in Fig. 2(a). Consequently, a classical spin component $J_{y,z}$ appears.

Although the ground state $|F=2⟩$ has five Zeeman-sublevels, the optical pumping process pumps most atoms (97.9%, as mentioned before) to $|F=2,m_F=-2⟩$, which makes the other three sublevels ($m_F=\mathrm{0,1,2}$) negligible. Therefore, the whole system can be simplified to a three-level $\mathrm{\Lambda }$-system. Spin dynamics in such a $\mathrm{\Lambda }$-configuration can be described by the master equation, $$\begin{gathered} \frac{\mathrm{d}\rho }{\mathrm{d}\mathit{t}}+\frac{1}{2}\{{\mathrm{\Gamma }}_{\mathrm{rel}},\rho \}=-\frac{i}{\hslash }[H,\rho ]+{\mathrm{\Gamma }}_{\mathrm{exc}}\rho , \\ H=-\hslash \left[\begin{array}{ccc}0& 0& \frac{{\mathrm{\Omega }}_1^{*}(t)}{\sqrt{3}}\\ 0& \mathrm{\Delta }& \frac{{\mathrm{\Omega }}_2^{*}(t)}{\sqrt{6}}\\ \frac{{\mathrm{\Omega }}_1(t)}{\sqrt{3}}& \frac{{\mathrm{\Omega }}_2(t)}{\sqrt{6}}& \mathrm{\Delta }\end{array}\right], \\ {\mathrm{\Gamma }}_{\mathrm{rel}}=\mathrm{Diag}(\mathrm{0,0,}\mathrm{\Gamma }), \\ {\mathrm{\Gamma }}_{\mathrm{exc}}\rho =\mathrm{Diag}(\frac{\mathrm{\Gamma }}{2}{\rho }_{33},\frac{\mathrm{\Gamma }}{2}{\rho }_{33},0), \end{gathered}$$(1)where $\rho$ is the density matrix, and ${\mathrm{\Omega }}_1$ and ${\mathrm{\Omega }}_2$ are the time-dependent Rabi frequencies of $\pi$ and ${\sigma }^{-}$ polarized light, respectively, with the irrational numbers in $H$ being the Clebsch–Gordan coefficients. Due to the falling edge of the pump laser, ${\mathrm{\Omega }}_{\mathrm{1,2}}$ monotonically decrease to zero. The two-photon detuning $\mathrm{\Delta }$ equals the frequency difference between the two ground states, which is the Larmor frequency induced by the bias magnetic field $B$ in the Zeeman sublevel scheme. In our experiment, $\mathrm{\Delta }$ is time-independent. $\mathrm{\Gamma }$ is the decay rate of the excite state $|3⟩$, phenomenologically set as the Doppler-broadened width. We have compared the calculation results from this simplified model with that of a more complete model, including Doppler broadening (integration over velocity distributions), multiple Zeeman levels, and ground state spin decay (see below), and found relatively small differences in the trends of the theoretical curves.

In the time-evolution problem considered, we assume that the initial state is the steady state prepared before the falling edge of the pump laser pulse, and then one can numerically calculate the state evolution from the master equation. In our model, we employ the following parameters: ${\mathrm{\Omega }}_1(0)=2\pi \times 7.09\mathrm{MHz}$, ${\mathrm{\Omega }}_2(0)=2\pi \times 35.5\mathrm{MHz}$, $\mathrm{\Gamma }=2\pi \times 500\mathrm{MHz}$, $\mathrm{\Delta }=2\pi \times 500\mathrm{kHz}$. For comparison with the experiment, we emulate the falling edge of the pump laser pulse with the Rabi frequency of a Gaussian function of time $t$, ${\mathrm{\Omega }}_{\mathrm{1,2}}(t)={\mathrm{\Omega }}_{\mathrm{1,2}}(0)\sqrt{\alpha (t)}={\mathrm{\Omega }}_{\mathrm{1,2}}(0)e^{-t^2/4{\tau }^2}$. Figure 2(b) shows the numerical results under the conditions of nonadiabatic sharp falling edge (red line, $\tau =1\mathrm{\mu s}$) and adiabatic slow falling edge (blue line, $\tau =20\mathrm{\mu s}$), respectively. In the adiabatic case, the coherence ${\rho }_{12}$ vanishes as the pump fully shuts off. In contrast, for the nonadiabatic case, the coherence remains finite, even when the pump laser is completely off. Such remaining coherence will be detected by the subsequent QND-probe pulse, hindering the preparation of the spin-squeezed state, as we show next.

To gain physical intuitions, we can also obtain analytical results after some approximations. First, we set the initial state to be $|1⟩$, because the $\pi$ polarization component is much smaller than the ${\sigma }^{-}$ component (${\mathrm{\Omega }}_1\ll {\mathrm{\Omega }}_2$). We can also neglect the population change, $${\rho }_{11}={\rho }_{11}^{(0)}=1,{\rho }_{33}={\rho }_{33}^{(0)}=0,{\rho }_{22}={\rho }_{22}^{(0)}=0,{\rho }_{32}={\rho }_{32}^{(0)}=0.$$(2)The master equation can be simplified to $$\begin{gathered} {\dot{\rho }}_{21}\begin{array}{c}=-{\mathrm{\Gamma }}_{12}{\rho }_{21}+i\frac{{\mathrm{\Omega }}_2^{*}}{\sqrt{6}}{\rho }_{31}\hfill \end{array}, \\ {\dot{\rho }}_{31}=-{\mathrm{\Gamma }}_{13}{\rho }_{31}+i\frac{{\mathrm{\Omega }}_2}{\sqrt{6}}{\rho }_{21}+i\frac{{\mathrm{\Omega }}_1}{\sqrt{3}}, \end{gathered}$$(3)where ${\mathrm{\Gamma }}_{12}=-i\mathrm{\Delta }\text{and}{\mathrm{\Gamma }}_{13}=\mathrm{\Gamma }/2-i\mathrm{\Delta }$.

Since ${\rho }_{31}$ adiabatically follows the ground-state coherence ${\rho }_{21}$, we can set ${\dot{\rho }}_{31}=0$. Then the master equation can be further simplified to $${\dot{\rho }}_{21}(t)=-({\mathrm{\Gamma }}_{12}+A_{22}·\alpha (t)){\rho }_{21}(t)-A_{12}·\alpha (t),$$(4)where $A_{12}={\mathrm{\Omega }}_{10}{\mathrm{\Omega }}_{20}^{*}/3\sqrt{2}{\mathrm{\Gamma }}_{13},A_{22}={|{\mathrm{\Omega }}_{20}|}^2/6{\mathrm{\Gamma }}_{13}$, and its initial state is the steady state ${\rho }_{21}(0)=-A_{12}/({\mathrm{\Gamma }}_{12}+A_{22})$.

To check the validity of the above simplified model, we numerically study a multilevel system where the Doppler velocity integration was formally considered. Two hyperfine levels $5S_{1/2},F=2\to 5P_{1/2},F^{\prime }=2$, with all 10 Zeeman sublevels are included. The excited state decay rate is set to be $\mathrm{\Gamma }=2\pi \times 6\mathrm{MHz}$ instead of the $\mathrm{\Gamma }=2\pi \times 500\mathrm{MHz}$ in the simplified model and the result is integrated over velocities of the atoms in the cell that obey Maxwell’s velocity distribution law. The master equation is $$\frac{\mathrm{d}\rho }{\mathrm{d}\mathit{t}}+\frac{1}{2}\{{\mathrm{\Gamma }}_{\mathrm{rel}},\rho \}=-\frac{i}{\hslash }[H,\rho ]+{\mathrm{\Gamma }}_{\mathrm{exc}}\rho ,$$$$H=\hslash \left[\begin{array}{cccccccccc}2\mathrm{\Delta }& 0& 0& 0& 0& \frac{{\mathrm{\Omega }}_1^{*}}{\sqrt{3}}& -\frac{{\mathrm{\Omega }}_2^{*}}{\sqrt{6}}& 0& 0& 0\\ 0& \mathrm{\Delta }& 0& 0& 0& 0& \frac{{\mathrm{\Omega }}_1^{*}}{2\sqrt{3}}& -\frac{{\mathrm{\Omega }}_2^{*}}{2}& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& -\frac{{\mathrm{\Omega }}_2^{*}}{2}& 0\\ 0& 0& 0& -\mathrm{\Delta }& 0& 0& 0& 0& -\frac{{\mathrm{\Omega }}_1^{*}}{2\sqrt{3}}& -\frac{{\mathrm{\Omega }}_2^{*}}{\sqrt{6}}\\ 0& 0& 0& 0& -2\mathrm{\Delta }& 0& 0& 0& 0& -\frac{{\mathrm{\Omega }}_1^{*}}{\sqrt{3}}\\ \frac{{\mathrm{\Omega }}_1}{\sqrt{3}}& 0& 0& 0& 0& -\delta +\frac{2}{3}\mathrm{\Delta }& 0& 0& 0& 0\\ -\frac{{\mathrm{\Omega }}_2}{\sqrt{6}}& \frac{{\mathrm{\Omega }}_1}{2\sqrt{3}}& 0& 0& 0& 0& -\delta +\frac{1}{3}\mathrm{\Delta }& 0& 0& 0\\ 0& -\frac{{\mathrm{\Omega }}_2}{2}& 0& 0& 0& 0& 0& -\delta & 0& 0\\ 0& 0& -\frac{{\mathrm{\Omega }}_2}{2}& -\frac{{\mathrm{\Omega }}_1}{2\sqrt{3}}& 0& 0& 0& 0& -\delta -\frac{1}{3}\mathrm{\Delta }& 0\\ 0& 0& 0& -\frac{{\mathrm{\Omega }}_2}{\sqrt{6}}& -\frac{{\mathrm{\Omega }}_1}{\sqrt{3}}& 0& 0& 0& 0& -\delta -\frac{2}{3}\mathrm{\Delta }\end{array}\right],$$$${\mathrm{\Gamma }}_{\mathrm{rel}}=\mathrm{Diag}(\gamma ,\gamma ,\gamma ,\gamma ,\gamma ,\gamma +\mathrm{\Gamma },\gamma +\mathrm{\Gamma },\gamma +\mathrm{\Gamma },\gamma +\mathrm{\Gamma },\gamma +\mathrm{\Gamma }),$$$$\begin{gathered} {\mathrm{\Gamma }}_{\mathrm{exc}}\rho =\mathrm{Diag}(\frac{\gamma }{5}+\frac{1}{3}\mathrm{\Gamma }{\rho }_{(2^{\prime },1),(2^{\prime },1)}+\frac{2}{3}\mathrm{\Gamma }{\rho }_{(2^{\prime },2),(2^{\prime },2)},\frac{\gamma }{5}+\frac{1}{2}\mathrm{\Gamma }{\rho }_{(2^{\prime },0),(2^{\prime },0)}+\frac{1}{6}\mathrm{\Gamma }{\rho }_{(2^{\prime },1),(2^{\prime },1)}+\frac{1}{3}\mathrm{\Gamma }{\rho }_{(2^{\prime },2),(2^{\prime },2)}, \\ \frac{\gamma }{5}+\frac{1}{2}\mathrm{\Gamma }{\rho }_{(2^{\prime },-1),(2^{\prime },-1)}+\frac{1}{2}\mathrm{\Gamma }{\rho }_{(2^{\prime },1),(2^{\prime },1)},\frac{\gamma }{5}+\frac{1}{3}\mathrm{\Gamma }{\rho }_{(2^{\prime },-2),(2^{\prime },-2)}+\frac{1}{6}\mathrm{\Gamma }{\rho }_{(2^{\prime },-1),(2^{\prime },-1)}+\frac{1}{2}\mathrm{\Gamma }{\rho }_{(2^{\prime },0),(2^{\prime },0)}, \\ \frac{\gamma }{5}+\frac{2}{3}\mathrm{\Gamma }{\rho }_{(2^{\prime },-2),(2^{\prime },-2)}+\frac{1}{3}\mathrm{\Gamma }{\rho }_{(2^{\prime },-1),(2^{\prime },-1)},0,0,0,0,0). \end{gathered}$$(5)Here, the base of the matrix is $\{|F=2,m_F=2⟩,|F=2,m_F=1⟩,|F=2,m_F=0⟩,|F=2,m_F=-1⟩,|F=2,m_F=-2⟩,|F^{\prime }=2,m_F=2⟩,|F^{\prime }=2,m_F=1⟩,|F^{\prime }=2,m_F=0⟩,|F^{\prime }=2,m_F=-1⟩,|F^{\prime }=2,m_F=-2⟩\}$, and the notation ${\rho }_{(i^{\prime },j),(k^{\prime },l)}$ means density matrix element between $|F^{\prime }=i,m_F=j⟩$ and $|F^{\prime }=k,m_F=l⟩$. $\delta$ represents the Doppler shift, $\gamma$ is a generic loss rate of atoms from all energy levels, accounting for collisions with the cell wall or other atoms. Other coefficients have the same meaning as those in the three-level model.

The multilevel numerical simulation was performed using the Atomic Density Matrix (ADM) package [19,20], where the following parameters are used: ${\mathrm{\Omega }}_1(0)=2\pi \times 7.09\mathrm{MHz},{\mathrm{\Omega }}_2(0)=2\pi \times 35.5\mathrm{MHz},\mathrm{\Gamma }=2\pi \times 6\mathrm{MHz},\gamma =2\pi \times 100\mathrm{Hz}$. The Doppler shift $\delta$ obeys the Gaussian distribution with half-width at half-maximum (HWHM): $2\pi ·\frac{1}{2}\sqrt{\frac{8\ln (2)kT}{{\mathit{m}}_A{c}^2}}\frac{c}{\lambda }$, where $k$ is the Boltzmann constant, $c$ is the speed of light, $T=326.65\mathrm{K}$ is the temperature of the atoms, $m_A=1.44\times {10}^{-25}\mathrm{kg}$ is the mass of a rubidium atom, and $\lambda =795\mathrm{nm}$ is the wavelength of light. The falling edge of the pump laser pulse obeys the same Gaussian function of time, ${\mathrm{\Omega }}_{\mathrm{1,2}}(t)={\mathrm{\Omega }}_{\mathrm{1,2}}(0)\sqrt{\alpha (t)}={\mathrm{\Omega }}_{\mathrm{1,2}}(0)e^{-t^2/4{\tau }^2}$.

To fully characterize the effect of the falling edge of the pump laser beam, we record the residual coherence ${\rho }_{12}(t)$ at $t=0.1\mathrm{ms}$ in the numerical simulation (where $t=0$ is the start of the falling edge) for different falling time $\tau$. As shown in Fig. 3(a), the residual ${\rho }_{12}$ decreases as the falling time $\tau$ increases. It is found that when $\tau$ is longer than 5 μs, the process is sufficiently “adiabatic” and the residual coherence nearly vanishes. We also investigate the dependence on the two-photon detuning $\mathrm{\Delta }$ and find that a smaller $\mathrm{\Delta }$ corresponds to a longer falling time that fulfills the adiabatic condition.

Although we performed full numerical calculation, to gain more intuition, analytical expressions can be obtained with some approximations. First, we assume that the time interval between the pump and probe pulse is long enough to let the pump laser power reduce to zero, which means $\alpha (t)\approx 0^{+}$. Second, we consider the case where falling time of the pump pulse is small enough compared to the ground-state coherence decay rate, i.e., $|{\mathrm{\Gamma }}_{12}\tau |\ll 1$. This is satisfied in our experiment. Therefore, we can solve ${\rho }_{21}(t)$ from the differential equation ${\rho }_{21}(\alpha \to 0^{+})\approx -\frac{\sqrt{2}{\mathrm{\Omega }}_{10}}{{\mathrm{\Omega }}_{20}}·\frac{1}{i\mathrm{\Delta }\tau +1}$, which clearly shows that (1) a longer falling time $\tau$ causes less residual coherence; and (2) given a larger detuning $\mathrm{\Delta }$, a shorter falling time $\tau$ is required to eliminate the residual coherence.

The influence of the repump laser applied to eliminate atomic populations in $|F=1⟩$ can also be explained by the model aforementioned, whose falling edge will also create an unwanted coherence and a transverse spin component. In the experiment, the Fourier frequency component in the falling edge of the pump laser creates the coherence between $|F=2,m_F=-2⟩$ and $|F=2,m_F=-1⟩$. It is detected as a dc signal, since the excited transverse spin oscillates at the Larmor frequency ${\mathrm{\Omega }}_L$, the same as the demodulation frequency of the lock-in amplifier. In contrast, the repump laser creates the coherence between $|F=1,m_F=0⟩$ and $|F=1,m_F=-1⟩$. The Zeeman splitting of these two states is different from that of $|F=2⟩$ (${\mathrm{\Omega }}_L$) [17] by ${\mathrm{\Delta }}_L$ (roughly proportional to the total magnetic field, about 2 kHz for ${\mathrm{\Omega }}_L=500\mathrm{kHz}$), leading to an oscillation signal with frequency of ${\mathrm{\Delta }}_L$ at the lock-in output. Due to experiment imperfections, there are unavoidable random fluctuations on the amplitudes of these unwanted coherences, causing extra noise in the detection of the quantum noises of the atomic spin.

The scattered dots in Fig. 3(a) show the measured amplitude of the unwanted $J_z$ component mapped onto the optical $S_y$ component, and it decreases for slower turn-off of the pump laser. This agrees qualitatively with our theoretical analysis of the adiabatic process. The lock-in output has an oscillation signal at frequency ${\mathrm{\Delta }}_L$, attributed to the beat (via the off-resonant Faraday interaction, i.e., the QND probing process) between the Zeeman coherences within the $F=2$ and $F=1$ states. When comparing signal amplitudes for different ${\mathrm{\Omega }}_L$, the lock-in amplifier’s frequency response to different ${\mathrm{\Delta }}_L$ must be taken into account. Alternatively, we can measure such a signal at ${\mathrm{\Delta }}_L$ by replacing the lock-in with a spectrum analyzer, which gives well-separated signal peaks at ${\mathrm{\Omega }}_L$ and ${\mathrm{\Omega }}_L\pm {\mathrm{\Delta }}_L$. The signal amplitude [$y$ axis of Fig. 3(a)] is the averaged peak height at ${\mathrm{\Omega }}_L\pm {\mathrm{\Delta }}_L$ derived from 500 repetitions of the pulse sequence, and the error bar is the standard deviation of five independent experiments.

We also study the dependence on $\mathrm{\Delta }$ by tuning the Larmor frequency ${\mathrm{\Omega }}_L$ through the bias magnetic field; the results are illustrated in Fig. 3(a). The dots show that the amplitude of the unwanted $J_z$ component decreases for longer falling time of the pump field, and the dashed line represents the numerical simulations. We can see that their trends agree qualitatively. Comparing Fig. 3(b) with Fig. 3(a), we can see that the results from the multilevel simulation with Doppler velocity integration are similar to those from the simplified model. Both can fit the experimental data well.

How does the transverse spin adversely affect spin squeezing? It is expected to add an extra signal to the mean value of the measured signal. We change the spectrum analyzer in Fig. 1 to a lock-in amplifier to obtain the time signal at the Larmor frequency. As shown in Fig. 4(a), an oscillation at 2 kHz appears in the mean value of the signal for the sharp falling edge. The oscillation is produced by atomic spin in $F=1$, as mentioned above. The contribution from $F=2$ should follow similar trends, but its amplitude is difficult to measure, as it appears near dc and is hard to distinguish from the spin PN. For comparison, we show the theoretical result in Fig. 4(b), where, following Ref. [21], we can get $⟨S_y^{\mathrm{out}}⟩=C|{\rho }_{(2,-2),(2,-1)}|+C^{\prime }|{\rho }_{(1,-1),(1,0)}|·e^{i{\mathrm{\Delta }}_{L}t}$. Here $⟨S_y^{\mathrm{out}}⟩$ is the mean value of our $S_y$ measurement, $C$ and $C^{\prime }$ are coefficients and $C^{\prime }\ll C$ because of the large detuning with respect to the hyperfine level $F=1$. ${\mathrm{\Delta }}_L$ is the Larmor frequency difference between two ground hyperfine levels $F=1$ and $F=2$. In addition, the transverse spin caused by the sharp falling edge will also increase the variance of the measured signal, as shown in Fig. 4(c), because there are inevitable residual systematic classical instabilities in the experiment. This extra signal will be slightly different for each repetition of the experiment. The larger this extra signal is, the more extra noise is added, contaminating the quantum noise detection for the atomic spin. Figure 4(d) shows the measured atom PN, and it can be seen that the noise power decreased by nearly 3 dB after using a slow falling edge. Our optimal spin squeezing using the conventional forward prediction QND protocol is about 2.3 dB [9] (less than 3 dB), which indicates that without the adiabaticity technique, the spin-squeezing signal will be completely buried in the classical noises.

We demonstrated the technique of adiabatic pulse control for CSS preparation in spin squeezing of large atom ensembles. By engineering the pulse shape of the optical pumping, we eliminated the classical noise induced by the sharp falling edge of pump lasers, which is the dominating noise source preventing spin squeezing in large atom number systems. This technique should be applicable to a wide range of quantum metrology and quantum information experiments involving ensembles of large atom number and/or large volume, such as the spin exchange relaxation free magnetometer [22].

Acknowledgment. We thank M. Balabas and Precision Glassblowing (Colorado, U.S.) for assistance in the vapor cell fabrication, and K. Mølmer for helpful discussions.