It is well known that the orbital angular momentum (OAM) of photons was discovered by Allen et al. [1] in 1992. Since the topological charge ℓ can be any integer, the OAM-carrying beams have countless orthogonal eigenstates, which allows them to have high-dimensional characteristics [2]. Benefiting from such high-dimensional characteristics, current applications of free-space optical (FSO) communication with OAM states are widely studied in the lab and real urban environments [3–10]. Naturally, the recognition of OAM modes in the receiving unit is one of the most important tasks for an optical communication system. A Gaussian mode converted by a forked hologram from the target OAM mode is the only mode that couples efficiently into the single-mode fiber [11]. More complicated computational holograms can be designed for the recognition of OAM superposition states [12]. Leach et al. presented the cascading additional Mach–Zehnder interferometers with dove prisms, which can sort OAM eigenstates into different paths [13]. By employing the Cartesian to log-polar transformation, one can convert the helically phased light beam corresponding to OAM state into a beam with a transverse phase gradient, and separate OAM eigenstates into different lateral positions [14–17]. Extensive research has also been carried out to characterize OAM modes by letting beams form diffraction patterns by passing through well-designed masks, such as multipoint interference [18], triangular aperture diffraction [19], angular-double-slit interference [20], and gradually changing-period grating [21].

However, all the above OAM mode-recognizing methods require a complicated optical alignment process for FSO communication. Generally, the OAM of a light beam depends on the choice of the reference axis [22]. A pure OAM eigenstate will transform into the superposition of OAM states in a displaced coordinate frame [23] and result in the mixing of information between adjacent modes. In the standard approaches to FSO communication with the polarization of photons, the transmitting and receiving units with a shared reference frame are required. In 2012, Ambrosio et al. implemented the quantum communication [24] with hybrid polarization-OAM-entangled states, and this proposal is rotation-immune to the shared reference frame. Displacement of the reference frame also imposes serious obstacles to the application of FSO communication with OAM states. To overcome these obstacles, misalignment correction is implemented by using the mean square value of the OAM spectrum as an indicator [25]. However, OAM spectrum measurement with high precision under the case of misalignment is necessary before the OAM spectrum correction.

Recently, with the rapid increase in computing power, deep learning (DL) [26] has once again become a hot topic in various disciplines. Trained deep neural networks (DNNs) show state-of-the-art performance in imaging through scattering media [27–29], phase retrieval [30,31], structure light recognition [9,32–36], and creating new quantum experiments [37]. A milestone in the history of convolutional neural networks (CNNs) is the appearance of ResNet proposed by He et al. [38]. The core of the ResNet model is to establish shortcuts or skip connections between early layers and later layers, which helps in the backpropagation of the gradient during the training process, so as to train a deeper CNN. As a representative of CNNs, DenseNet [39] performs well in the ImageNet data set, it establishes dense connections between all early layers and later layers, and, specifically, each layer accepts all preceding layers as its additional inputs. Another major characteristic of DenseNet is the feature reuse through the connection of features on the channel. These characteristics allow DenseNet to achieve better performance than ResNet with fewer parameters and computational costs.

In this work, we implement an alignment-free fractal multipoint interferometer for hyperfine OAM mode recognition assisted by DL. By using a well-designed fractal multipoint mask (FMM) to sample the complex phase fronts of OAM modes, wealthy diffraction intensity patterns can be recorded for different OAM modes. Meanwhile, the diffraction patterns are stable against reference misalignment because of the inherent periodic structure of the FMM. Stochastic disturbances of three different parameters of the OAM states are set in the experiments: (i) beam waist size ω∈[0.45,0.55]mm; (ii) initial phase of OAM states φ0∈[0,2π]; (iii) lateral translation range along the x and y directions Δx,Δy∈[−0.25,0.25]mm. Here, the maximum lateral displacement of the FMM we added with respect to the perfectly on-axis beam is about ±0.5 beam waist size along the x and y directions, respectively. With the above three parameters changing randomly at the same time, we implement the recognition of OAM eigenstates with an accuracy of 100%. Adjacent OAM superposition states with a Bures distance (BD) close to 0.01 are also recognized with an accuracy higher than 98.3%. Benefiting from the simple FMM configuration and superhigh resolution recognition with high accuracy, our detection method is very useful for systems where the optical vortices are expected to be on very large scales, such as in FSO communication [9] and astronomical optical vortices [40].

The LG modes have a complex field amplitude given byLGpℓ(r,φ)∝rℓLpℓ(2r2ω2)exp(−r2ω2)exp(−iℓφ),(1)where (r,φ) are the radial and azimuthal coordinates, respectively. ω is the beam waist, and Lpℓ(2r2/ω2) is the associated Laguerre polynomial. ℓ is the topological charge, and p is the radial mode index. The complex amplitude field after the FMM isU(x,y)=∑n=1Ncirc[(x−xn)2+(y−yn)2r0]LGpℓ(r,φ),(2)where circ(x,y) is the transmittance function of the aperture in the FMM, and r0 is the radius of the FMM aperture. xn, yn are the central coordinates of the nth aperture. In the experiments, we implement the lateral translation misalignment by adding random lateral displacement Δx and Δy along the x and y directions, respectively, for the FMM. Here we employ a model for the pattern of florets in the head of a sunflower proposed by Helmut Vogel [41] in 1979 to arrange the position of each aperture, which is given byxn=C1ncos(2πC22n),yn=C1nsin(2πC22n),(3)where C1 is the constant scaling factor, and C2 is the divergence angle. Considering the Fraunhofer limit, the far-field intensity pattern in the detector plane I is given by the Fourier transform of the field in the FMM plane, I∝|F[U(x,y)]|2,(4)and F[·] represents the Fourier transform.

To estimate the topological charge of the LG modes with the recorded intensity patterns I, we define I=H(ℓ), where the H(·) represents the forward physical process that produces the diffraction pattern from the incident LG mode with the topological charge ℓ. The optimization problem can be implicitly written asℓ^=argminℓL{H(ℓ),I,R(ℓ)},(5)where ℓ^ is the estimate of the inverse, ℒ is the objective function to minimize, and R(ℓ) is the regularizer, or prior knowledge term that imposes constraints on the solution.

We first perform the DenseNet-121 to recognize LG eigenstates with topological charge ℓ∈{−5,−4,⋯,5} and p=0. In order to test the robustness of the proposed method, stochastic disturbance of three parameters of the OAM states is set simultaneously for the acquisition of each diffraction intensity pattern: (i) beam waist size ω∈[0.45,0.55]mm; (ii) initial phase of OAM states φ0∈[0,2π]; (iii) lateral translation range along the x and y directions Δx,Δy∈[−0.25,0.25]mm. A total of 1100 experimental diffraction intensity patterns and their corresponding topological charge ℓ as labels are used as the data set, with 100 samples for each topological charge ℓ. All 1100 samples are randomly shuffled, of which the first 850 samples are used as the training set; the remaining 250 samples never participate in the training process.

In conclusion, we experimentally implemented hyperfine OAM mode recognition under strong misalignment by using an alignment-free fractal multipoint interferometer assisted by DL. The misalignment includes three stochastic disturbances of parameters: (i) beam waist size ω∈[0.45,0.55]mm; (ii) initial phase φ0∈[0,2π]; (iii) lateral translation Δx,Δy∈[−0.25,0.25]mm. Here, the maximum lateral misalignment of the FMM we added with respect to the perfectly on-axis beam is about ±0.5 beam waist size along the x and y directions, respectively. The well-tuned DenseNet-121 is demonstrated to be robust for recognizing very similar superposition states with a small BD of 0.01 between adjacent modes under the above strong misalignment. Benefiting from the robustness of the proposed method and simple FMM configuration, this scheme shows potential application for FSO communication where the optical vortices are expected to be on a large scale and the misalignment between the transmitting and receiving units is inevitable.