With the rapid development of the Internet, IoT, 5G communications, and other technologies, the global demand for data transmission is exploding. In the face of the physical transmission capacity limitations of single-mode fiber (SMF), space division multiplexing (SDM) provides a new direction for the optical transmission system [1–3]. The core of SDM lies in the development of the fiber’s lateral spatial dimensions, which may extend the spatial channel and thus enhance the transmission capacity of a single fiber. There are multiple forms of SDM such as multi-core multiplexing, mode division multiplexing (MDM), and their combination [4–7]. Among them, MDM mainly focuses on exploiting the orthogonality of fiber modes and can be implemented with the few-mode fiber (FMF) or multimode fiber (MMF). However, one drawback of MDM is that it is greatly affected by mode coupling, which makes it highly dependent on partial or full multiple-input multiple-output (MIMO) digital signal processing (DSP) [8–11]. Therefore, directly deploying MDM for short-distance optical interconnection requires the application of coherent optical communications and MIMO, the high cost of which is usually unacceptable. To address this problem, mode group (MG) multiplexing based on weakly coupled fibers has been developed, which enables the use of intensity modulation and direct detection (IMDD) techniques in short-distance scenarios [12–17]. As the group delay of multiple modes within the same MG is almost equal, these modes can be simultaneously detected and regarded as a single spatial channel. Although some mode freedom is sacrificed, the complexity of the MDM system can be greatly reduced in this way.

In the MG multiplexing system, two essential devices are the multiplexer and demultiplexer, which are used to excite and separate the MG channels. As the modes within a same MG are strongly coupled, it is feasible to excite one or more of them. Such multiplexing approaches include the photonic lantern [18,19], mode selective coupler (MSC) [20,21], inverse-designed microchip [22], optical coordinate transformer [23,24], multiplane light conversion (MPLC) [25–28], or diffraction optical network (DON) [29,30]. The effectiveness of these methods has been extensively studied, but to develop an MG demultiplexer, the above schemes cannot be directly applied in reverse. The reason is that all modes in MG need to be received simultaneously; otherwise it will cause fluctuations in the received energy. One possible solution is to receive the modes separately and then add them together, but this comes at the expense of more detections [13,31]. Alternatively, the demultiplexed modes can be further combined into a single FMF or MMF, so that the MG can be collected thoroughly with one detector [16,32]. A simpler solution is to directly couple the MGs into the corresponding fibers, which has been achieved by degenerate MSC [33,34], MPLC devices [15–17,35], and Fabry–Perot thin-film filters [36]. Among them, the MPLC devices have unique advantages in mode number, loss, and crosstalk. However, the previous design strategies have not fully exploited the mode field freedom supported by the MMF. As a result, a large number of phase plates are required to realize MG demultiplexing, and the conversion efficiency cannot be maximized.

In this paper, benefiting from the huge light manipulation freedom, we proposed and demonstrated an MG demultiplexing scheme based on the DON. The DON has a similar architecture to MPLC but differs in design philosophy. One of the advances of the DON is that it uses error back-propagation to build the network, which allows it to achieve more sophisticated tasks [37,38]. In the field of light manipulation, its applications include optical mode manipulation [39], orbital angular momentum (OAM) mode switching [40], OAM multiplexing holography [41], OAM spectrum measurement [42], OAM mode add–drop multiplexer [43], and diffractive waveguide design [44]. Following a similar technical route, this work further applies DON to the separation of OAM mode groups. To receive the energy of the entire MG, we utilized an MMF array to couple the output fields of the DON. Considering the large mode field freedom of MMF, a non-deterministic mode conversion strategy is proposed here to optimize the DON. In this way, the mode conversion efficiency and coupling efficiency to the MMF can be maximized. Meanwhile, the number of the required phase plates can be significantly reduced. Following this solution, we then designed a 6-MG demultiplexer which contains only five phase plates. The average loss and crosstalk of the demultiplexer are 0.5 dB and $-25\mathrm{dB}$, respectively. The device was then fabricated using three times binary lithography and measured with a reflective experimental setup. After being coupled to the MMF array, an average insertion loss of 4.5 dB and an inter-mode crosstalk of $-18\mathrm{dB}$ were achieved. These results imply great application prospects of the scheme.

An MG demultiplexing model using DON is constructed by several cascaded phase plates and spatial diffraction, as shown in Fig. 1(a). The model’s input is collimated from a weakly coupled OAM fiber that contains several radial first-order OAM MGs, and the fiber parameters can be seen in Ref. [45]. One benefit of this fiber is that the number of modes within an MG does not change with the order, which is conducive to the upgrade of MG demultiplexing scale. In addition, although the MG discussed here uses the OAM mode, it is equivalent to use the linearly polarized (LP) mode. After undergoing the DON, the OAM modes are divided into several MGs and coupled to the corresponding MMF. Here MMF is used for energy reception instead of SMF; because multiple orthogonal modes are contained within the same MG, an SMF cannot collect the energy from all states simultaneously. Unlike the case of SMF reception, the target output of the DON is non-determinstic and can be represented by arbitrary superposition of the mode bases of MMF. This approach can significantly enhance the degree of freedom of the target output, thereby improving the conversion efficiency of DON and reducing the number of required phase plates. But, in essence, the DON still functions as a mode converter. The detailed implementation of this scheme will be further discussed below.

In order to optimize the parameters of the DON, it is necessary to establish a cascade diffraction propagation process of the DON. For simplicity, only the phase plate parameters are considered here. From the Rayleigh–Sommerfeld diffraction, the propagation of the beam in the DON can be viewed as the following linear process: $$E_{N+1,m}=E_{0,m}{F}_1{M}_1{F}_2{M}_2\dots F_N{M}_N{F}_{N+1},$$(1)where $N$ denotes the number of phase plates, $E_{0,m}$ and $E_{N+1,m}$ denote the $m$th input and the corresponding output light field, $M_k=\mathrm{diag}[\exp (j{P}_k)]$ is a diagonal matrix that denotes the phase plate $P_k$, and $F_k$ denotes the transfer matrix of the $k$th diffraction. $P_k$ is the variable of DON to be optimized, and $F_k$ depends on the spatial position of these phase plates. The linear model here is mainly used to facilitate the description and understanding of the characteristics of light beam transmission, while the specific calculation of light beam diffraction is given by the angular spectrum diffraction (ASD) method.

After obtaining the output of the network, the traditional approach is to directly compare the output with a specific target spatial mode to evaluate the performance. This is suitable in the case of SMF reception, because the mode field of SMF is unique and the conversion efficiency is the overlap integral of the output and the SMF field. In the case of MMF reception, the mode conversion is undetermined. Accordingly, we need to project the output field onto all mode bases of MMF as follows: $$c_{m,n,r}=⟨E_{N+1,m},B_{n,r}⟩,r\le S,$$(2)where $c_{m,n,r}$ represents the projection coefficient, $B_{n,r}$ represents the $r$th mode basis of the $n$th receiving MMF channel, and $S$ is the number of mode bases. So the coupling efficiency is the sum of all mode basis components, and the transfer matrix $T$ can be calculated as follows: $$T_{i,j}={\sum }_{\text{all}m\in {\mathrm{MG}}_i}\left({\sum }_{r=1}^S{|c_{m,j,r}|}^2\right)/|{\mathrm{MG}}_i|,$$(3)where ${\mathrm{MG}}_i$ represents the set of modes belonging to the $i$th MG, and $|{\mathrm{MG}}_i|$ is its cardinality. To quantify the performance of the DON, the loss function can be defined as follows: $$L=1-{\sum }_{n=1}^Q{T}_{n,n}/Q,$$(4)where $L$ represents the average value of coupling loss and $Q$ represents the total number of MGs. To further optimize crosstalk and bandwidth, the above loss function needs to include their effects with certain weights. To minimize $L$, the parameter gradient of the phase plates can be determined based on the error back-propagation method: $$\frac{\partial L}{\partial H_k}=\frac{-\lambda }{2\pi }\mathrm{Im}\left\{{\sum }_{m=1}^Q\mathrm{diag}\right[{(E_{0,m}{F}_1{M}_1\dots F_k{M}_k)}^T(\frac{\partial L}{\partial E_{N+1,m}}{F}_{N+1}^T{M}_N\dots F_{k+1}^T)\left]\right\},$$(5)where $H_k$ represents the thickness distribution of the $k$th phase plate, $\lambda$ represents the working wavelength, and diag represents taking the diagonal elements of the matrix.

The above gradient information indicates the optimization direction, from which the phase plate parameters can be further iteratively updated using the gradient descent method. The core operations in the phase plates optimization are organized into a flowchart as shown in Fig. 1(b). Moreover, based on the optimized transmission matrix, the averaged insertion loss and inter-MG crosstalk are calculated as follows: $${\mathrm{IL}}_{\mathrm{ave}}=10{\log }_{10}\left(\frac{1}{Q}{\sum }_{n=1}^Q{T}_{nn}\right),{\mathrm{XT}}_{\mathrm{ave}}=10{\log }_{10}\left(\frac{1}{Q}{\sum }_{n=1}^Q\frac{1}{Q-1}{\sum }_{m\ne n}^Q{T}_{mn}/T_{nn}\right).$$(6)

To illustrate the effectiveness of the above scheme, the MG demultiplexing performance based on deterministic mode conversion and non-deterministic mode conversion using different numbers of phase plates is shown in Fig. 1(c). The detailed simulation configuration can be found in Appendix A. The topological charge of the incident OAM modes is $l=-5$ to $l=+5$, and the outputs are coupled to an MMF array with a pitch of 250 μm. For deterministic mode conversion, the modes with $l<0$ are converted to LP11a, the modes with $l>0$ are converted to LP11b, and the mode $l=0$ is converted to LP01. For non-deterministic mode conversion with six bases or ten bases, we selected the first six or ten mode bases (LP01, LP11a, LP11b, LP21a, LP21b, LP02, LP31a, LP31b, LP12a, and LP12b) of MMF, respectively. It can be seen from Fig. 1(c) that using ten mode bases does not significantly reduce the mode conversion loss relative to six mode bases. Meanwhile, the divergence angle of the high-order mode may exceed the capacity of the microlens, so six mode bases were used in the real experiment. But, obviously, the use of a non-deterministic mode conversion strategy has great advantages in both reducing the number of phase plates and improving conversion efficiency. With non-deterministic mode conversion, approximately 0.5 dB loss can be achieved with only five phase plates, while deterministic mode conversion requires ten phase plates to achieve similar performance.

Next, we further illustrate the MG demultiplexing process with non-deterministic mode conversion, as shown in Fig. 2. We used five phase plates here to achieve sufficient efficiency and minimize experimental complexity. The optimized depth distributions are shown in Figs. 2(P1)–2(P5). The size and spacing of the phase plates are consistent with the previous simulation. These phase plates are all rectangular, but the phase of some areas around them is set to 0. Since the light passing through the surroundings is extremely weak, the operation has little effect on the performance of the device and can greatly reduce the exposure time during processing. The MG demultiplexing process based on the DON is shown in Figs. 2(a1)–2(a7), where the intensity distribution is a superposition of all channels. To better show the transformation process of the light beam, the low-energy part of the intensity distribution is replaced with a white background. The input OAM mode field distribution is determined by the FMF. The outputs are divided into six different spatial positions corresponding to six MGs. The light field transformations of the modes $l=+5$ and $l=-5$ are shown in Figs. 2(b1)–2(b7) and Figs. 2(c1)–2(c7), respectively. We can see that the DON can well demultiplex the OAM MGs, and the corresponding transmission matrix is shown in Fig. 2(d). All MGs have a demultiplexing efficiency of more than 88%. Furthermore, we also show the loss and crosstalk in the C-band in Fig. 2(e). It can be seen that the demultiplexing scheme is also compatible with wavelength division multiplexing. With six mode bases, the loss in the full C-band is better than 0.8 dB, and the crosstalk is better than $-22\mathrm{dB}$. In contrast, the case with deterministic mode conversion based on five phase plates shows significantly worse performance. It is important to note that using 10 mode bases shows greater crosstalk; this is due to the increased ability to receive unwanted light. Since the real MMF has more mode bases, the simulated crosstalk is underestimated to some extent.

Based on the above simulation, the phase plate is discretized into an eight-level depth distribution to be processed by binary fabrication. The device processing flow can be found in Appendix B. We then built an experimental setup as shown in Fig. 3. Above the picture are the enlarged details of the device. From the cross markers, it can be seen that the phase plates are composed of three overlapping patterns. The alignment error between them is less than 0.7 μm. In order to reduce the adjustment dimensions of phase plates, a folded reflective optical path was adopted in the measurement. The input beams are the OAM modes emitted by FMF, which are excited and switched by a spatial light modulator (SLM, HOLOEYE PLUTO). These OAM fields are close to the Lagurre-Gaussian (LG) beam, and the mode field diameter is about 600 μm. The input beams then pass through the cavity formed by the phase plates and mirror and are demultiplexed to different locations. On the output side, there are two different setups for free-space optical detection and characterization after coupling into MMF. In the free space, the output beams are imaged by a lens and then observed using an infrared camera (Hamamatsu C12741-03). Further quantitative characterization requires coupling the beam into MMF array via an microlens array (MLA). Before coupling, the alignment of the MMF array and the MLA was first done, and then they were bonded together using UV glue to form a module.

After completing the installation for free-space observation, we sequentially switched the OAM mode from the $-5$th to the $+5$th order and recorded the light field with the camera. The incident OAM mode fields from the FMF are shown in Fig. 4(a). Without affecting the contrast of the picture, the faint portion with less than 1% of the brightest part is replaced with white background. Since the modes are randomly coupled inside the optical fiber, the emitted optical field is no longer a pure OAM mode. As shown in Fig. 4(b), most of the coupling occurs within the MG, and the OAM beams with the same absolute value of TC are output to the same position. All modes are well output to uniformly distributed locations corresponding to their MGs, and the crosstalk between different MGs appears to be small. A video is provided in the supporting document showing observations from the camera when switching between different OAM modes (see Visualization 1).

The free space results show good performance of the device, and we then further couple the output to an MMF array to quantitatively evaluate the demultiplexing. As shown in the real device in Fig. 4(c), we used a 2 m OM3 MMF array to couple the output to evaluate the loss and crosstalk of the MG demultiplexer. The output power of the MMF array is detected by a spatial optical power meter (Thorlabs PM100d). After averaging multiple modes within an MG, the power transmission matrix of the MG demultiplexer is shown in Fig. 4(c). The relative power shown in the figure is the ratio of the MMF output to the FMF input optical energy. Accordingly, the loss ranges from 4.1 to 4.9 dB, with an average of 4.5 dB. The loss of phase plates is about 2.6 dB, and the loss caused by coupling to the MMF is about 1.9 dB. The inter-MG crosstalk is better than $-12\mathrm{dB}$, with an average of $-18\mathrm{dB}$.

The results show good feasibility of the MG demultiplexing scheme and have the potential to handle mode groups (see Appendix C). To further improve the results in the paper and reduce the differences between simulation and experiments, there are still some possible directions for improvement. (1) Since eight discrete phases are used here, this results in an additional diffraction loss of about 1.1 dB. Using more phase steps or adopting grayscale exposure [46] will directly reduce this loss. (2) Due to the large number of reflections, increasing the reflectivity of the phase plates and the mirror will further improve the efficiency. (3) The coupling loss may also be improved by reducing the losses of the MMF and microlens array. (4) The discrepancy between experimental and simulation results may also come from the quality of the collimated OAM beam and the alignment of the experimental setup. Nevertheless, the main point of this work lies in its unique strategy, which greatly reduces the difficulty of MG demultiplexing. As shown above, higher performance can be achieved with fewer phase plates, which will reduce the error caused by the experimental implementation.

In summary, we proposed and demonstrated a high-efficiency MG demultiplexing scheme for short-distance SDM. This approach breaks the convention of using fixed targets in the design and instead employs non-deterministic mode conversion. Based on this strategy, the energy efficiency can be significantly improved. We first verified this scheme in simulation and designed a set of phase plates based on the error back-propagation calculation method. The simulation results show that for the demultiplexing of six OAM MGs, only five phase plates are required to achieve the desired loss and crosstalk. Then the phase plates were processed through the multiple overlay processes, and a reflective experimental system was built to evaluate the performance of the device. The results show that the average loss from FMF output to MMF array reception is about 4.5 dB, and the average crosstalk between different MGs is about $-18\mathrm{dB}$. These results fully demonstrate the effectiveness of our scheme and are expected to be further applied in the short-range SDM communications.