Atmospheric turbulence (AT) severely affects the transmission of vortex beams (VBs) transmitted in the atmosphere. Wavefront distortion, coherence destruction, and orthogonality destruction of multiplexed VBs are the main effects of AT, which directly increase crosstalk among channels and reduce communication performance. To improve the robustness of optical orbital angular momentum (OAM) communications, considerable efforts have been made to effectively compensate for the phase distortion of VBs. The adaptive optical method is widely used but requires multiple iterations and complicated hardware that is not affordable or easily operated by most researchers, causing tremendous difficulties for further study. Recently, taking advantage of powerful signal processing techniques, deep learning has been widely used in many fields such as image classification and optical communication, providing researchers with a new approach for addressing these problems. In this study, we propose a novel method of AT compensation based on a deep learning method to effectively correct the distorted composite Bessel-Gaussian (BG) vortex beam and improve the robustness of OAM multiplexing communication.

Using a deep learning method, we designed a new model called the phase extraction network (PhaNet), which combines a residual network with a feature pyramid for AT phase extraction (Fig. 2). The PhaNet model can automatically learn the mapping relationship between the intensity distribution of the distorted beam and the turbulence phase under different orbital angular momenta. It contains seven convolutional layers, four residual layers, six deconvolution layers, and three feature fusion layers. A total of 96000 images of BG vortex beam intensity with a specified turbulence range were randomly generated, 80000 of which were used as training data, with the remaining 16000 serving as test data. Following training with the loads of the studied samples, the PhaNet model was used to directly predict AT phase screens based on the intensity distribution of the distorted composite BG vortex beam. The turbulence phase can be compensated by loading a reverse-predicted phase into the received composite BG vortex beam. We then used the AT compensation system as a physical model (Fig. 5) to study the AT effect compensation of a composite BG vortex beam by mode purity and intensity correlation coefficient under different conditions of turbulence intensities and distances.

To predict the entire turbulence phase, we successfully constructed the PhaNet model, which requires the intensity distribution of the distorted beam as input. Comparison results (Fig. 3) show that in the 21-layer structure, the mean loss significantly decreases, whereas the iterations show an inverse trend. When the number of iterations is 4000, the training loss reaches a plateau at 0.00957521. Therefore, to ensure the effectiveness of the predicted results in the PhaNet model, we chose 80000 training data and 4000 iterations as the training conditions. If better prediction performance is required, the amount of training data or number of iterations must be further increased. However, increasing the number of input samples increases the computational cost and lengthens the prediction time. To verify the generalization ability of the PhaNet model, we used the previously trained model to perform turbulence compensation for the composite BG beam propagating under different conditions by changing the parameters of turbulence (Fig. 6) and distances (Fig. 7), and we then analyzed the mode purity and intensity correlation coefficient. Under the conditions in which the topological charges l₁=4 and l₂=10 propagate from strong to weak turbulence and the composite BG beam has a 1000 m transmitting distance, the mode purity of l₁=4 increases by approximately tenfold, from 3.41%, 3.54%, and 4.61% to 30.70%, 31.21%, and 31.35%, respectively, after compensation. Simultaneously, the mode purity of l₂=10 shows a similar trend, increasing remarkably from 6.09%, 6.23%, and 6.30% to 64.68%, 65.45%, and 66.53%, respectively, after compensation. In addition, the beam intensity correlation coefficient of l₂=10 combined with different topological charges l₁ increases from 0.4199, 0.4596, and 0.5281 to 0.97 and greater. The mode purity of the composite BG beam (l₁=4 and l₂=10) at a propagation distance of 1500 m in strong turbulence are 3.50% and 2.43% respectively, which can be improved to 30.55% and 64.07%, respectively, after compensation. The beam intensity correlation coefficient of l₂=10 combined with different topological charges l₁ increases from 0.6477 and 0.3495 to 0.9794 and 0.9268, respectively.

We propose an AT compensation scheme for a composite BG vortex beam based on a phase extraction network. The compensation effect of the phase extraction network on a distorted composite BG vortex beam under different turbulence intensities and propagation distances is numerically analyzed. The results show that after phase compensation, when the composite BG vortex beam propagates 1000 m under different turbulence intensities, the intensity correlation coefficient can be increased to greater than 0.97, whereas the intensity correlation coefficient is improved to 0.9268 when the transmission distance increases to 1500 m under strong turbulence. These results show that the PhaNet model possesses a good generalization ability for quickly and accurately predicting the equivalent turbulent phase screen, even in unknown turbulence environments, and thus has great potential for improving the performance of OAM communication.