A high-capacity quantum channel is the most prominent task for future practical quantum communication [1]. Recently, much research has been focused on high-dimensional entanglement [2], which is proposed to meet the growing demand for information capacity per photon and to implement high-capacity quantum tasks, such as quantum key distribution [3,4], quantum teleportation of multiple degrees [5,6], and quantum computation [7].

Parallel to the discrete variable high-dimensional entanglement of single photons, entanglement between copropagating quantum modes in one beam, named continuous-variable (CV) multimode entanglement [8], has been proposed for high-capacity quantum tasks. To date, preparing multimode entanglement with individual degrees of freedom (DoFs), such as spatial mode [9], frequency [10,11] or wavelength [12], and time bin [13], has gradually matured. With the boost of multimode entanglement, multiplex CV quantum communication has been realized [14] and demonstrates the prospect of reinforcing the channel capacity [15,16]. Their work mostly concentrated on single DoF preparation. The research on 2-DoFs entanglement has been carried out and is at their early stages [17,18]. So far, the experimental demonstration on the 3-DoFs entanglement for CVs has not to our knowledge been presented. However, the infancy task with more than two available DoFs is more interesting and rewarding. With the merit of diverse physical DoFs entanglement, it not only can multiply increase the data capacity [19], but also have potential advantages, such as better efficient information processing [20], better security, and enhanced robustness to noise [21,22]; furthermore, it may also realize special quantum tasks, such as quantum multiparameter measurement [23,24], quantum interfaces between DoFs [25,26], and especially deterministic high-dimension quantum teleportation [5,27,28]. How to create such CV high-dimensional entanglement will be a challenging question.

The optical parametric oscillator (OPO) is the workhouse [29] for many quantum laboratories to generate the CV quantum resource, and up to now, it is an indispensable device in CV quantum information. Nearly all works for studying entanglement generation by OPO are limited to one or two DoFs, such as frequency comb-type entanglement [15,30], including femtospectral entanglement [31], and spatial mode entanglement among Hermite–Gaussian (HG) modes [32–34] or Laguerre–Gaussian (LG) modes [35], or even hyperentanglement [17] of polarization and HG modes. Merging more than two available DoFs during the OPO process is challenging to realize the multimode resonance simultaneously, yet appealing, because it is not only able to generate CV high-dimensional entanglement, but also can be used to construct large-scale cluster states for CV quantum computation [36].

Here, we report the direct generation of a new type CV high-dimensional entanglement with 3-DoFs [frequency, polarization, and orbital angular momentum (OAM)] based on a type-II OPO by compensating for the astigmatism between spatial modes and dispersion between frequency modes. We experimentally verify the 3-DoFs entanglement by the 14 pairs of HG mode entanglement measurements and demonstrate that the output has stricter CV hyperentanglement. Then, the channel capacity enhanced quantum dense coding (QDC) communication has been demonstrated by utilizing the entanglement resource. With such a new type of quantum source, it has the potential to conduct crucial missions, for example, deterministic quantum teleportation of a single photon with multiple degrees [5,28] and alignment-free satellite-based CV quantum communication [37]. Moreover, it is easy to apply to fiber-based quantum communication as eigenmodes in optical fibers.

Photon pairs produced via a type-II parametric down-conversion process have many accessible DoFs that can be exploited for the production of multiple DoFs entanglement simultaneously as illustrated in Fig. 1(a), which could possess three attributes, polarization ($H$, $V$), OAM (with topology charge $+l,-l$), and frequency ($\pm n\mathrm{\Omega },n=0,1,2,\dots$). In the OPO process [Fig. 1(b)], the polarization between a pair of correlated photons ($A$, $B$) is orthogonal to each other, and the angular momentum $(l_A+l_B=0,l_A=0,\pm 1,\pm 2,\dots ,l_B=0,\mp 1,\mp 2,\dots )$ and energy $({\omega }_A+{\omega }_B={\omega }_p)$ satisfy the conservation, where ${\omega }_p$ is the frequency for the pump photon. The down-converted photons have 3-DoFs correlation at the same time. Due to the frequency mode selection of the OPO cavity, the output appears as a frequency comb structure $({\omega }_A=\omega \pm n\mathrm{\Omega },{\omega }_B=\omega \mp n\mathrm{\Omega })$, where $\omega$ is the center frequency of the down-converted photons, $\mathrm{\Omega }$ is the free spectrum range (FSR) of the OPO, and $n$ is the order of the frequency sideband. In each frequency sideband, there are four correlated pairs.

The interaction Hamiltonian of the system is expressed by $$\begin{gathered} {\widehat{H}}_{\mathrm{int}}=i\hslash \sum _{n,l}{G}_{n,l}({\widehat{a}}_{H,+l,+n\mathrm{\Omega }}^{+}{\widehat{b}}_{V,-l,-n\mathrm{\Omega }}^{+}+{\widehat{a}}_{V,+l,+n\mathrm{\Omega }}^{+}{\widehat{b}}_{H,-l,-n\mathrm{\Omega }}^{+} \\ +{\widehat{a}}_{H,-l,+n\mathrm{\Omega }}^{+}{\widehat{b}}_{V,+l,-n\mathrm{\Omega }}^{+}+{\widehat{a}}_{V,-l,+n\mathrm{\Omega }}^{+}{\widehat{b}}_{H,+l,-n\mathrm{\Omega }}^{+})+\mathrm{H}.\mathrm{C}., \end{gathered}$$(1)where ${\widehat{a}}^{+}$ and ${\widehat{b}}^{+}$ are photon creation operators of down-converted photons $A$ and $B$, the subscripts $H$, $V$ denote horizontal and vertical polarization modes and $\pm l$ for OAM modes, and the parameter $G_{n,l}$ regulates the interaction strength in the $n$th-order frequency sideband with topological charge $l$. It explicitly indicates a frequency comb entanglement, spin angular momentum (SAM), and OAM entanglement simultaneously of the system.

Since all frequency sidebands have the same Hamiltonian form, we take the first order as an example. From the perspective of OAM or SAM, the corresponding Hamiltonian has the form $${\widehat{H}}_{\mathrm{SAM}}=i\hslash G_{1,l}({\widehat{a}}_H^{+}{\widehat{b}}_V^{+}+{\widehat{a}}_V^{+}{\widehat{b}}_H^{+})+\mathrm{H}\text{.C}.,$$(2)$${\widehat{H}}_{\mathrm{OAM}}=i\hslash G_{1,l}({\widehat{a}}_{+l}^{+}{\widehat{b}}_{-l}^{+}+{\widehat{a}}_{-l}^{+}{\widehat{b}}_{+l}^{+})+\mathrm{H}\text{.C}.$$(3)

The Hamiltonian is similar to the ones for two-photon polarization, leading to suppression of fluctuation in certain Stokes observables in OAM and SAM, which indicate the entanglement of these attributes [38]. In the CV domain, it can be used to test Bell-type correlations.

Due to the spatial selection of the OPO cavity, the spatial modes are output with the same order. For the case of $l=1$, the entangled beam can be verified through the complete entanglement measurement of the first-order HG mode denoted with $h(H{G}_{10})$ and $v(H{G}_{01})$; see Fig. 1(c).

The experimental setup is illustrated in Fig. 2(a). The multimode OPO is driven by the LG pump mode to generate 3-DoFs entanglement and is then detected by BHD with the spatial and frequency tailored local field. The OPO consists of two KTP crystals ($3\mathrm{mm}\times 3\mathrm{mm}\times 5\mathrm{mm}$) and a mirror. The side crystal KTP1 with a wedge angle of 1° (with high reflectivity at 540 nm and 1080 nm) is used for nonlinear interaction, and the mirror with a radius of curvature of 50 mm (with transmission of 10% at 1080 nm and transmission of 20% at 540 nm) serves as the output coupler of the entangled beams at 1080 nm.

The most determinative technology is multimode resonance simultaneously in an OPO since nonlinear crystals have dispersion for frequency modes and astigmatism for higher-order transverse modes; thus, frequency sidebands and higher-order transverse modes cannot be naturally output at the same resonance condition. These modes inevitably suffer from resonance detuning if not well compensated. We first conduct a theoretical analysis and obtain the optimal detuning condition (${\mathrm{\Delta }}_A=-{\mathrm{\Delta }}_B=\mathrm{\Delta }$); see Appendix A. Then, we turn to compensation. The Gouy phase between spatial modes can be compensated for with the combination of a half-wavelength plate and crystal [17], but the frequency gap between the frequency sideband modes is worsened and expands gradually as the frequency order increases (about $40n$ MHz). For the case of the first-order frequency sideband, the gap is close to our cavity bandwidth (about 46 MHz), which greatly limits the quality and spectral range of the attainable entanglement resources. Here, as shown in Fig. 2(b), the middle crystal KTP2 is oriented to make the crystal $Z$ axis perpendicular to KTP1 to compensate for the dispersion to guarantee the near resonance of all frequency sideband modes (about $n$ kHz). The price to pay is the detuning for HG modes ($a_{Hh},a_{Hv},a_{Vh},a_{Vv}$). However, we can benefit form the inverse discrepancy of cavity length between $\mathrm{H}{\mathrm{G}}_{01}$ and $\mathrm{H}{\mathrm{G}}_{10}$ modes [$L_{Hh}-L_{Hv}=-(L_{Vh}-L_{Vv})\ne 0$], which is supported by the experimental observation in Figs. 7(b1) and 7(b2). With comprehensive consideration of the high-order transverse mode astigmatism, cavity bandwidth, and nonlinear interaction strength, we set the crystal length to 5 mm. At last, we conduct experiments under an optimal entanglement condition (${\mathrm{\Delta }}_{Hh(Vv),\pm n\mathrm{\Omega }}=-{\mathrm{\Delta }}_{Vh(Hv),\mp n\mathrm{\Omega }}=\mathrm{\Delta }$) with uniform detunings about 10 MHz for each first-order HG mode by controlling the KTP temperature independently and locking the OPO with a 45º linear polarization $\mathrm{H}{\mathrm{G}}_{10}^{45°}$ seed beam.

To liberate the attainable entanglement levels, we prepare the pump field $\mathrm{L}{\mathrm{G}}_1^0$, which has higher pump conversion efficiencies than the fundamental mode and avoids oscillation. The 540-nm fundamental mode beam passes through the FQ-PM and a cavity (Filter) to produce pure $\mathrm{H}{\mathrm{G}}_{11}$; then, $\mathrm{L}{\mathrm{G}}_1^0$ is formed through an MC composed of two cylindrical lenses. Furthermore, a wedge-angle crystal (KTP1) is used to compensate for the optical length of 540 nm to realize the simultaneous resonance of the pump field and the down-converted field (triple resonance), aiming to obtain sufficient pump power.

For entanglement certification, the corresponding local field ($\mathrm{L}{\mathrm{O}}_H\&\mathrm{L}{\mathrm{O}}_V$) is generated. Arbitrary sideband frequencies ($\omega \pm n\mathrm{\Omega },\mathrm{\Omega }=2.387\mathrm{GHz}$) are generated by loading radio frequency on the FAM. One local field with positive sideband frequencies ($\omega +n\mathrm{\Omega }$) is filtered by a high finesse (about 1000) mode cleaner. The other local field ($\omega -n\mathrm{\Omega }$) is obtained in the same way. In the spatial domain, the first-order HG mode is prepared via a TQ-PM and purified by the mode cleaner. Different spatial modes ($\mathrm{H}{\mathrm{G}}_{10}\&\mathrm{H}{\mathrm{G}}_{01}$) can be switched handily with a DP. Limited by the frequency bandwidth of the local setup, we measured the entangled state of the first fourth-order frequency sideband.

As shown in Fig. 3, we measured the quantum correlations with the 3-DoFs measurement basis. There are two correlated pairs on the central frequency due to frequency mode degeneracy [I.a. (${\widehat{a}}_{Hh,\omega }$, ${\widehat{b}}_{Vh,\omega }$), I.b. (${\widehat{a}}_{Hv,\omega }$, ${\widehat{b}}_{Vv,\omega }$)]. On each frequency sideband except the central one, there are 8 modes forming 4 correlated pairs, totaling 12 correlated pairs for three frequency sidebands [II.a (${\widehat{a}}_{Hh,\omega +\mathrm{\Omega }}$, ${\widehat{b}}_{Vh,\omega -\mathrm{\Omega }}$), II.b (${\widehat{a}}_{Vh,\omega +\mathrm{\Omega }}$, ${\widehat{b}}_{Hh,\omega -\mathrm{\Omega }}$), II.c (${\widehat{a}}_{Hv,\omega +\mathrm{\Omega }}$, ${\widehat{b}}_{Vv,\omega -\mathrm{\Omega }}$), II.d (${\widehat{a}}_{Vv,\omega +\mathrm{\Omega }}$, ${\widehat{b}}_{Hv,\omega -\mathrm{\Omega }}$), III.a (${\widehat{a}}_{Hh,\omega +2\mathrm{\Omega }}$, ${\widehat{b}}_{Vh,\omega -2\mathrm{\Omega }}$), III.b (${\widehat{a}}_{Vh,\omega +2\mathrm{\Omega }}$, ${\widehat{b}}_{Hh,\omega -2\mathrm{\Omega }}$), III.c (${\widehat{a}}_{Hv,\omega +2\mathrm{\Omega }}$, ${\widehat{b}}_{Vv,\omega -2\mathrm{\Omega }}$), III.d (${\widehat{a}}_{Vv,\omega +2\mathrm{\Omega }}$, ${\widehat{b}}_{Hv,\omega -2\mathrm{\Omega }}$), and IV.a (${\widehat{a}}_{Hh,\omega +3\mathrm{\Omega }}$, ${\widehat{b}}_{Vh,\omega -3\mathrm{\Omega }}$), IV.b (${\widehat{a}}_{Vh,\omega +3\mathrm{\Omega }}$, ${\widehat{b}}_{Hh,\omega -3\mathrm{\Omega }}$), IV.c (${\widehat{a}}_{Hv,\omega +3\mathrm{\Omega }}$, ${\widehat{b}}_{Vv,\omega -3\mathrm{\Omega }}$), IV.d (${\widehat{a}}_{Vv,\omega +3\mathrm{\Omega }}$, ${\widehat{b}}_{Hv,\omega -3\mathrm{\Omega }}$)]. The top subfigure box [Figs. 3(a) and 3(b)] and bottom subfigure box [Figs. 3(c) and 3(d)] in each frequency sideband section are correlation measurements for the $\mathrm{H}{\mathrm{G}}_{10}$ and $\mathrm{H}{\mathrm{G}}_{01}$ modes, respectively. The left column [Figs. 3(a) and 3(c)] and right column [Figs. 3(b) and 3(d)] in each subfigure box are for horizontal and vertical polarization modes with positive sideband frequencies. The top column [Figs. 3(a1)–3(d1)] and bottom column [Figs. 3(a2)–3(d2)] in each subfigure box show squeezing for the sum of amplitude quadrature ${\mathrm{\Delta }}^2{\widehat{X}}_{ij,n}^{+}={\mathrm{\Delta }}^2({\widehat{X}}_{i,\pm n\mathrm{\Omega }}^A+{\mathrm{\Delta }}^2{\widehat{X}}_{j,\mp n\mathrm{\Omega }}^B)$ and the difference of phase quadrature ${\mathrm{\Delta }}^2{\widehat{Y}}_{ij,n}^{-}={\mathrm{\Delta }}^2({\widehat{Y}}_{i,\pm n\mathrm{\Omega }}^A-{\mathrm{\Delta }}^2{\widehat{Y}}_{j,\mp n\mathrm{\Omega }}^B)$ [$i=Hh,Vh,Hv,Vv$; $j=Vh,Hh,Vv,Hv$; $\widehat{X}=(\widehat{a}+{\widehat{a}}^{+})/\sqrt{2}$; $\widehat{Y}=i({\widehat{a}}^{+}-\widehat{a})/\sqrt{2}$; $n=\mathrm{0,1,2,3}$], having the squeezing level around $-2.60\pm 0.13\mathrm{dB}$. Since the total measurement efficiency is $0.84\pm 0.02$, the inferred squeezing is $-3.34\pm 0.13\mathrm{dB}$. It should be noted that all measurements satisfy the criteria of Duan et al. [39] and Simon [40] ${\mathrm{\Delta }}^2{\widehat{X}}_{ij,n}^{+}+{\mathrm{\Delta }}^2{\widehat{Y}}_{ij,n}^{-}<1.15<2$, clearly showing the existence of 14 first-order HG pairs of entanglement and frequency comb entanglement in the output. Limited by the phase-matching bandwidth of the OPO [41], we estimate that approximately 8000 entanglement pairs containing 2000 frequency pairs can be excited in our system.

According to Eqs. (2) and (3), there are SAM and OAM entanglements in the first-order frequency sideband. For $l=1$, we have $$\begin{gathered} I({\widehat{S}}_1,{\widehat{S}}_3)=\frac{{\mathrm{\Delta }}_{A-B}^2{\widehat{S}}_1+{\mathrm{\Delta }}_{A+B}^2{\widehat{S}}_3}{2|[\delta {\widehat{S}}_1,\delta {\widehat{S}}_3]|}=0.54\pm 0.01<1, \\ I({\widehat{O}}_1,{\widehat{O}}_3)=\frac{{\mathrm{\Delta }}_{A+B}^2{\widehat{O}}_1+{\mathrm{\Delta }}_{A-B}^2{\widehat{O}}_3}{2|[\delta {\widehat{O}}_1,\delta {\widehat{O}}_3]|}=0.54\pm 0.01<1. \end{gathered}$$(4)

${\widehat{S}}_k\text{and}{\widehat{O}}_k(k=\mathrm{1,2,3})$ are the Stokes operators of SAM and OAM, respectively. Here, it is a faithful CV hyperentanglement of OAM and SAM, which has not been demonstrated, according to the stricter sufficient but not necessary criterion of entanglement for OAM [42] and SAM [43]. There is similar hyperentanglement on the other frequency sidebands. Therefore, the OPO output has high-dimensional CV entanglement states with frequency comb entanglement and OAM and SAM entanglement simultaneously.

QDC is a typical protocol in quantum information, one that pursues to transmit as much classical information as possible per qubit [44]. The CV QDC schemes were theoretically proposed [45,46] and experimentally demonstrated soon based on single-mode entangled light or one fundamental Gaussian mode entanglement [47–49], which introduces a lack of flexibility, and is detrimental to its scalability. The way forward is to deterministically encode the multimode within one beam [50,51]. Until recently, Shi et al. [15] experimentally demonstrated a practical channel multiplexing quantum dense coding (MQDC) communication by exploiting the extracted four pairs of entangled sideband modes. Chen et al. [16] focused on another important physical attribute (OAM) and demonstrated the substantially enhanced channel capacity of the OAM MQDC scheme. Their work mostly concentrated on single DoFs preparation, whereas, leaving the infancy task with more than two available DoFs.

As a demonstration, we experimentally conduct the conventional QDC scheme, 1-DoF MQDC scheme, and 2-DoFs MQDC scheme. For the conventional QDC scheme, the transmitted signals of the sender are encoded on single-mode $\mathrm{H}{\mathrm{G}}_{10}$. For the 1-DoF MQDC scheme, we encode the message simultaneously on the monochromatic HG superposition mode ($\mathrm{H}{\mathrm{G}}_{10},\mathrm{H}{\mathrm{G}}_{01}$). For the 2-DoFs MQDC scheme, the messages are simultaneously encoded on superposition modes (${\mathrm{HG}}_{10}^{\omega -\mathrm{\Omega }}$, ${\mathrm{HG}}_{10}^{\omega }$, ${\mathrm{HG}}_{10}^{\omega +\mathrm{\Omega }}$, ${\mathrm{HG}}_{01}^{\omega -\mathrm{\Omega }}$, $\mathrm{H}{\mathrm{G}}_{01}^{\omega }$, $\mathrm{H}{\mathrm{G}}_{01}^{\omega +\mathrm{\Omega }}$). The experimental setup is shown in Fig. 4. At the Alice station, the modes multiplexing schemes are switched by the mode element control of the auxiliary source. The frequency side modes are produced by loading radio frequency on the FAM. It can set single frequency componnents, two frequency components, or three frequency components by controlling the bias voltage of FAM. The multispatial mode can be obtained with a mode converter, which consisted of a vortex QP, an HWP, and a polarization beam splitter. For example, the HG QDC scheme utilizes monochromatic $\mathrm{H}{\mathrm{G}}_{10}^{45}$ as an auxiliary mode. Then, the auxiliary fields are encoded on amplitude and phase quadrature with AM and PM in which both the modulation powers are set equally. The well-prepared auxiliary field is coupled in phase with a previous shared entanglement state via a 90:10 beam splitter. Thus, at the Bob station, the amplitude and phase signals are extracted by BHD, respectively, with the aid of the other shared entanglement state.

The noise power spectra of each scheme, in Fig. 5, are measured at the same condition of average photon number $m$ in the channel. We repeat the measurement process with different $m$ where the same $m$ is represented with the same color scale. The black trace is the SNL. The blue traces are the amplitude quadrature sum and the phase quadrature difference without modulation, respectively. The signal peaks at 3.5 MHz are generated by modulation of the AM and PM at the Alice station. The signal-to-noise ratios required for deriving the corresponding channel capacities can be obtained from the noise power spectra.

We compare the three schemes mentioned above with several well-known schemes from the perspective of channel capacity. As demonstrated in Fig. 6(a), the circle dots with the error bars are calculated from the experimental data, and the smooth curves are from theoretical formulas in Refs. [16,46,49] where black and red solid lines are the channel capacities for the scheme of the coherent state ($C_{\mathrm{ch}}$) and the Fock state ($C_{\mathrm{Fock}}$). The solid pink line denotes the optimum channel capacity for single-mode dense coding ($C_{\mathrm{sd}}^{\mathrm{opt}}$). The blue lines represent several dense encoding schemes demonstrated in this paper where the light blue dotted line is the conventional QDC scheme ($C_{\mathrm{sd}}$), the blue dotted line is for the HG MQDC scheme ($C_{\mathrm{d}\mathrm{d}}$), and the dark blue dotted line is for the space-frequency MQDC scheme ($C_{\mathrm{xd}}$). The channel capacity of the conventional QDC scheme exceeds its corresponding coherent scheme for all values of $m$ ($C_{\mathrm{s}\mathrm{d}}>C_{\mathrm{c}\mathrm{h}}$), whereas, inferior to the absolute maximum channel capacity for a single-mode ($C_{\mathrm{s}\mathrm{d}}<C_{\mathrm{Fock}}$) at our entanglement levels. A further improvement of channel capacity is realized via channel multiplexing and optimizing entanglement [Fig. 6(b)]; when $m>2.99$ and $m>1.88$, $C_{\mathrm{d}\mathrm{d}}$, and $C_{\mathrm{s}\mathrm{d}}^{\mathrm{opt}}$ will exceed $C_{\mathrm{Fock}}$, respectively. By increasing the number of DoFs in the communication mode, the channel capacity of our space-frequency multiplexing scheme is substantially enhanced ($C_{\mathrm{x}\mathrm{d}}>C_{\mathrm{s}\mathrm{d}}^{\mathrm{opt}}$ when $m>1.46$).

By compensating both the astigmatism and the dispersion, we have directly generated high-dimensional CV entanglement states with frequency comb entanglement, OAM, and SAM entanglement simultaneously. We verified the existence of strict CV hyperentanglement, which has not been presented. Benefitting from the multiple DoFs entanglement, it can be utilized in high-dimensional quantum protocols. In particular, assisted by hybrid techniques, it was promising to implement both fundamental and applicable tasks, such as high-dimensional quantum teleportation of a single photon, Bell-inequality tests from CVs, and interfaces with matter and multiparameter measurements.

We have also experimentally demonstrated space-frequency MQDC for the first time, which shows the ability to enhance the channel capacity of quantum communication. In the proof of the principle experiment, the amplitude and phase quadrature signals of each channel mode were extracted per measurement one after another via homodyne measurement. It was feasible now to improvement with the spatial separation technique [24,52]. The full information can be extracted simultaneously with the heterodyne measurement where the orthogonal spatial modes and frequency modes were separated into spatially distinct beams with mode separators. On the other hand, the channel capacity can be further improved by increasing the squeezing parameter and the number of multiplexed modes in the channel since the information carrier modes were fully extendable.

To summarize with such a new type of quantum source, it no doubt enriched the quantum communication capacity, whereas, the hidden potential of the CV 3-DoFs entanglement as well as the open questions remains to be explored and exploited.