Vortex beams, endowed with orbital angular momentum (OAM), have captivated researchers for several decades. Pioneering work by Allen et al. in 1992 demonstrated that these beams [1] are characterized by an azimuthal phase term $\exp (il\phi )$ with $l$ the topological charge and $\phi$ the azimuthal angle, inspiring plenty of applications across diverse domains in both classical and quantum physics [2–8]. Capitalizing on the inherent high-dimensional orthogonality among OAM modes, a beam can carry a series of discrete, selective interval and power weight OAM modes, which are termed as OAM combs [9,10]. Such OAM comb offers significant potential for the transmission of vast amounts of information [11–14] and serves as a flexible key for photonics computing [15–18].

Previous research has introduced various schemes for generating OAM modes, where a single element, such as spiral phase plate or spatial light modulator, typically provides only one OAM mode, limiting scalability [19–21]. As the number of multiplexed OAM modes increases, the cost and complexity will rapidly grow and with the resultant required multiple elements [22–24], it is highly desirable to generate a large number of OAM modes in a simple, scalable, and cost-efficient way, leading to a consensus on the need for a phase-only modulation of OAM combs’ on-demand tailoring [11,25,26]. Different from generating single-mode OAM beams directly, creating OAM combs solely through superposing spiral phases is not feasible due to the inevitable mode intensity loss in phase-only modulation [27]. Current schemes, such as mode iteration [27], genetic algorithms [28], pattern-search strategies [29], and adaptive modification [30], have been employed to address this challenge. However, these schemes still face troubles such as initial set dependency, long iteration time, and uncertain convergence.

Inspired by deep neural networks with powerful abilities in extracting high-dimensional features [31–33], there is significant potential to establish an intelligent, data-driven framework for optimizing phase designs [34–36]. In this paper, we propose a multi-scale fusion learning U-shaped neural network (MSUNet) for on-demand tailoring of broadband OAM combs within a phase-only hologram. Our approach extracts high-dimensional features from the target OAM comb and compensates for mode intensity loss through multi-scale learning at the latent feature space, aiming to generate a high-dimensional OAM hologram. Notably, there are no ground-truth holograms employed in the training; instead, the scalar diffraction is incorporated in network design to calculate the modulated optical field, enabling an analysis of the OAM spectrum, forming a loss constraint. By this means, the issue of lacking ground-truth in supervised training is addressed, thus enhancing the network interpretability. Proof-of-principle experiments demonstrate that our proposal significantly achieves faster computational speed, higher modulation precision, and higher manipulation dimensionality, with a mode range of $-75$ to $+75$, an average root mean square error (RMSE) of 0.0037, and a fidelity of 85.01%, all achieved in about 30 ms. Furthermore, we utilize broadband OAM combs in optical convolution calculation, enabling vector convolution for arbitrary discrete functions with an accuracy of 0.0138 in RMSE and 82.82% in fidelity, showcasing the extended capability of our proposal. This study introduces a simple, fast, accurate, and scalable approach for on-demand broadband OAM comb tailoring, contributing to a wide range of advanced OAM-based technologies and applications.

Figure 1 presents a comprehensive conceptual framework for the intelligent tailoring of a broadband OAM comb and its application in optical convolution. This framework enables on-demand customization of OAM comb settings, including mode range, interval, and distribution, to form a target OAM spectrum. The corresponding target intensity and phase patterns of the target OAM comb serve as input data for the neural network training. The proposed MSUNet architecture consists of two main processes: feature extraction and feature fusion. In the feature extraction process, the input data patterns pass through a feature extractor, transforming into high-dimensional features, resulting in a feature map. Subsequently, feature fusion is performed using a multi-scale scheme. By weighting the high-dimensional intensity and phase features at multiple scales, the mode intensity loss caused by phase-only modulation can be compensated for, thereby refining the superposed hologram. Since there are no standard holograms available as ground-truth for training, we incorporate the physical process of scalar diffraction to calculate the optical field modulated by the output refined phase-only hologram, enabling an analysis of the OAM spectrum through a pre-trained deep residual network (DRN) [37]. The difference between this output OAM comb and the target is used as a constraint for back-propagation, addressing the lack of ground-truth in supervised training, enhancing the network interpretability, and achieving reliable network training.

Furthermore, such straightforward generation of OAM combs enables the application in optical convolution. The harmonic coefficients of the OAM comb, such as the OAM spectrum, can also be treated as discrete functions. Analogous to Fourier transformation, the vector convolution of any two discrete functions in the OAM domain is equivalent to the spectra of the product of their optical fields in the spatial domain. Therefore, the convolution of any two discrete functions [$F(l)$ and $G(l)$] can be obtained by measuring the spectrum of their optical field product. Given the advantages of a phase-only nature, the modulation result of the product of their optical fields can be achieved by simply adding the respective phase-only hologram of the OAM combs. This means that the convolution results of the two OAM combs [$F(l)*G(l)$] can be easily obtained by detecting the OAM spectrum of the final diffraction field, eliminating the need for complex and expensive convolution calculations. Notably, the calculation speed is at the speed of light. In other words, the proposed OAM comb tailoring scheme provides an efficient path for optical convolution calculation.

Figure 2(a) illustrates the feature extraction workflow, which mainly involves a feature extractor to transform the input target intensity and phase patterns into easily separable high-dimensional features at the latent feature space. Figure 2(b) details the structure of a U-shaped neural network [38], which serves as a feature extractor. This architecture consists of four sets of $2\times 2$ max-pooling layers and four sets of $2\times 2$ up-sampling layers. Each set of pooling and up-sampling layers is preceded by two $3\times 3$ convolutional layers with a rectified linear unit (ReLU) as the activation function, which is used to introduce nonlinear transformations in the feature extraction process. Copy and crop operations are employed between sampling layers with the same spatial dimensions in order to mitigate the issue of gradient explosion that can arise from deep feature representations. Finally, a $1\times 1$ convolutional layer is used to adjust the spatial dimensions. After the feature extraction process, a feature map is obtained, which represents the overall high-dimensional features of the input target OAM comb. In order to compensate for the mode intensity loss resulting from phase-only modulation, we introduced a multi-scale scheme that combines the target intensity and phase pattern feature maps by weighting the high-dimensional features at multiple scales; thus the superposed phases can be refined into a phase-only hologram for the target OAM comb. The multi-scale scheme [39] is expressed as follows: $${\varphi }_{\mathrm{holo}}=\alpha \sum _{i=1}^m{p}_i·f_{\mathrm{unet}}(I)+\beta \sum _{i=1}^m{q}_i·f_{\mathrm{unet}}(P),$$(1)where ${\varphi }_{\mathrm{holo}}$ represents the phase-only hologram; $\alpha$ and $\beta$ are hyper-parameters initially set as $\alpha =0.2$ and $\beta =0.8$, respectively. This initial values choice reflects a strategy to prioritize phase reconstruction while still considering intensity information, which is further optimized during network training. $p$ and $q$ denote scale factors, $m$ is the number of scaling layers, and $f_{\mathrm{unet}}(·)$ refers to the feature extractor. $I$ and $P$ represent the input intensity and phase pattern data, respectively.

Figure 2(c) displays the fusion process of intensity and phase high-dimensional features. Through the weighted combination of different-sized convolutionally sampled feature maps at varying scales, the phase-only hologram that includes intensity information compensation can be obtained. Specifically, as illustrated in Fig. 2(d), the multi-scale scheme includes three $2\times 2$ up-sampling layers, three $2\times 2$ down-sampling layers, and four $1\times 1$ convolutional layers. This module performs multi-scale feature extraction utilizing up-sampling layers to construct a feature pyramid, followed by spatial dimension transformation through the down-sampling layers. The final step involves feature fusion across multiple scales using the $1\times 1$ convolutional layers, which combine the features through weighted summation as a fusion output. Considering that there are no standard holograms available as ground-truth for training, we introduce the physical process of scalar diffraction, angular spectrum transmission, to obtain the output modulated optical field and then use a pre-trained DRN to analyze its output OAM spectra as a loss constraint. Such process reads $$\{{|a_l|}^2\}=f_{\mathrm{DRN}}(f_{\mathrm{AS}}({\varphi }_{\mathrm{holo}})),$$(2)where $a_l$ corresponds to the amplitude of OAM mode $|l⟩$, thus $\{{|a_l|}^2\}$, forming the OAM spectrum. The function $f_{\mathrm{DRN}}(·)$ refers to the process of the pre-trained DRN model and $f_{\mathrm{AS}}(·)$ represents the angular spectrum transmission. The difference between this output and the target OAM comb is used as a constraint for loss backward, addressing the lack of ground-truth in supervised training, enhancing the network interpretability, and achieving reliable network training. The RMSE is employed as a loss function to evaluate the difference: $$\text{Loss}=L^{\mathrm{RMSE}}(\{{|a_l|}^2\},\widehat{\{{|a_l|}^2\}})=\sqrt{\frac{1}{2n+1}\sum _{l=-n}^{+n}{(\{{|a_l|}^2\}-\widehat{\{{|a_l|}^2\}})}^2},$$(3)with $n$ the OAM mode range (see more details in Appendix A).

Proof-of-principle experiments are carried out to evaluate the performance of the proposed MSUNet for on-demand broadband OAM comb tailoring and its applications in optical convolution. The experimental setup is sketched in Fig. 3(a). A 1617 nm Gaussian beam emitted from a distributed feedback laser diode passes a half-wave plate (HWP) and a polarized beam splitter (PBS) in sequence, so as to obtain horizontally linear polarization to match the demand of phase-only modulation of the spatial light modulator (SLM, Holoeye, PLUTO-TELCO-013-C). The phase-only hologram obtained from MSUNet is encoded onto the SLM to generate the target OAM comb. After passing through a plano-convex lens with focal length 200 mm, the diffraction field is collected by an infrared CCD camera for subsequent OAM spectrum analysis.

Specifically, for tailoring OAM combs, we first input the target OAM comb intensity and phase patterns into the trained MSUNet on a computer to obtain the encoded phase-only hologram. This hologram is then encoded onto the SLM, and the intensity distribution of modulated optical fields can be observed at the CCD. For OAM spectrum measurement, due to the phase-only property, we can directly superpose the anti-spiral phase onto the hologram, resulting in a series of holograms with anti-spiral phases. By sequentially switching these holograms on the SLM, OAM spectrum measurement can be conducted (see more details in Appendix B). Similarly, for the efficient optical convolution, we can directly superpose the holograms from MSUNet for $F(l)$ and $G(l)$ to obtain the product optical field of $f(\phi )$ and $g(\phi )$. The same approach applies to OAM spectrum measurement. By sequentially superposing anti-spiral phases onto the superposed hologram and switching them in sequence on the SLM, the OAM spectra can be measured.

Considering that broadband OAM combs involve a large number of superposed OAM modes, after normalization, the absolute values of the relative intensity for each mode are small. RMSE may appear low due to these inherently small amplitude values, while this does not affect its effectiveness in training and reflecting results. However, it may not be particularly intuitive when presenting results. Therefore, we introduced an additional metric to provide a more straightforward evaluation of performance, while still including RMSE.

Given that evaluating multiple-mode OAM beams through mode purity is not feasible, we introduced the concept of a density matrix, analogous to concepts in quantum optics, to assess the quality of the output OAM comb. An OAM comb $|\mathrm{\Psi }⟩$ could simultaneously contain multiple OAM states as $$|\mathrm{\Psi }⟩=\sum _{l=-n}^{+n}{a}_l|l⟩,$$(4)where $|l⟩$ represents the OAM state. Thus, the density matrix $\rho$ for the OAM comb is defined as $\rho =|\mathrm{\Psi }⟩⟨\mathrm{\Psi }|$. By measuring the density matrix difference between the target and the experimental OAM combs, the similarity between the two OAM combs can be assessed—a smaller numerical difference indicates higher modulation quality. For better verification and quantitative evaluation, the fidelity $F$, which illustrates the similarity between the experimental result and the corresponding target OAM comb, is exploited here. The higher the fidelity, the better the alignment between two states. The fidelity can be calculated through the density matrix as $F=⟨{\mathrm{\Psi }}_{\mathrm{tar}}|{\rho }_{\exp }|{\mathrm{\Psi }}_{\mathrm{tar}}⟩$.

As an illustrative example, Fig. 3(b) shows the phase-only hologram and corresponding visualizations of the simulated and experimental optical field patterns for a 17-mode OAM comb with a minimum mode interval of one. Its OAM spectrum is measured and provided in Fig. 3(d), which displays the target OAM spectrum (blue bars) and the experimentally generated OAM spectrum (red bars). The corresponding density matrix difference is shown in Fig. 3(e). The MSUNet accurately tailors the OAM comb, achieving an RMSE of 0.0029 and a fidelity of 84.53%, within a testing time of 29 ms. Similarly, Fig. 3(c) presents the phase-only hologram and visualizations for a 36-mode OAM comb crossing mode range of $-75$ to $+75$. The measured OAM spectrum is shown in Fig. 3(f) and the corresponding density matrix difference is given in Fig. 3(g). For this case, the MSUNet achieves an RMSE of 0.0028 and a fidelity of 83.63%, within a testing time of 33 ms. All the experimental results align closely with simulations, demonstrating the accuracy of the MSUNet in tailoring broadband OAM combs. These numerical results strongly support the effectiveness of our proposal (see more details in Appendix C).

Since our proposal is available for tailoring broadband OAM combs that represent any discrete functions, we can further utilize OAM combs towards optical convolution calculation. Convolution is a fundamental mathematical operation widely used in signal and image processing. It produces a new function from two original functions, representing how a function is modified by the other. For OAM combs, similar to the product of functions in the spatial domain corresponding to the convolution of their Fourier coefficients, the OAM comb, with its rotational symmetry, can be expanded by Fourier transformation on azimuth into helical harmonics. In spherical coordinates, this expansion is represented by the superposition of multiple helical harmonic functions, forming the OAM spectrum [40]. Consequently, the optical fields product of OAM combs in the spatial domain corresponds to the convolution of the harmonic coefficients in the helical harmonic domain, where the OAM spectrum convolution is denoted as $$F(l)*G(l)=H(f(\phi )·g(\phi )),$$(5)where $H(·)$ denotes the helical harmonic transformation, $F(l)$ and $G(l)$ are the OAM spectra of OAM combs $f(\phi )$ and $g(\phi )$, respectively, and $H(f(\phi )·g(\phi ))$ represents the OAM spectra of the optical field product $f(\phi )·g(\phi )$ (see more details in Appendix D).

As illustrated in Fig. 4(a), for arbitrary functions (OAM combs) $F(l)$ and $G(l)$, the MSUNet can generate corresponding holograms ${\varphi }_f$ and ${\varphi }_g$ accurately. When a Gaussian beam passes through such two holograms in sequence, a diffraction field can be obtained, which is the product in spatial domain $f(\phi )·g(\phi )$ and is actually a new OAM comb. According to the aforementioned OAM convolution theory, the convolution result of $F(l)$ and $G(l)$ is the helical harmonic transformation of $f(\phi )·g(\phi )$, namely, the OAM spectrum of the OAM comb.

Proof-of-principle experiments confirm the practical operability, with example results shown in Figs. 4(b)–4(d). Figures 4(b) and 4(c) depict the original functions, OAM comb $F(l)$ and OAM comb $G(l)$, respectively. Instead of performing complex mathematic convolution calculations, our scheme provides a more efficient approach by simply measuring the OAM spectrum of a diffracted field that is modulated by the superposing holograms from the two original OAM combs to obtain the result. Figure 4(d) shows the experimental measurement results of the diffracted field and the vector convolution results of the two OAM combs $[F(l)*G(l)]$, with an RMSE of 0.0138 and fidelity of 82.82%. Such promising results proved that our approach significantly reduces the complexity and cost of convolution operation, highlighting its potential for extension to the advanced OAM-based applications.

This work presents an intelligent and efficient approach for on-demand tailoring of broadband OAM combs within a phase-only hologram, demonstrating in a highly efficient optical convolution. Our scheme overcomes the key limitations such as mode intensity loss in phase-only modulation of multiplexed OAM beams, enabling on-demand tailoring of further broadband combs with faster computational speed, higher modulation precision, and higher manipulation dimensionality. Proof-of-principle experiments, covering a variety of mode intervals and numbers of OAM modes ranging from $-75$ to 75, validate the effectiveness of our proposal, reaching an average RMSE of 0.0037 and fidelity of 85.01%. The whole process takes about 30 ms. Moreover, we utilize the broadband OAM combs in optical convolution operations, obtaining an accuracy of 0.0138 in RMSE and fidelity of 82.82%, demonstrating the extended capability of our proposal. The proposed approach, characterized by its simplicity, speed, accuracy, and scalability, opens new avenues for advanced applications of high-dimensional OAM beams in optical computing, laser manipulation, and other photonics-based technologies, paving the way for future innovations in the field.