It is well known that a light beam with a phase singularity nested in its wave front carries an orbital angular momentum (OAM) equivalent to $l\hslash$ per photon^[1]. Now, there is considerable interest in these unique properties arising from the special phase structures^[2]. The vortex beam (VB) has many applications in optical manipulation^[3], optical communications^[4], and quantum optics^[5]. Many researches have been devoted to the generation, transformation, or modulation of VBs via planar optical components, such as dielectric-based metasurface^[6], waveguide^[7], and nanoslits^[8].

The complicated phase pattern of light beam carrying OAM is of particular interest in modern optics^[9]. In this paper, we proposed a novel approach to detect the phase pattern of a VB by reflection on a symmetrical metal cladding waveguide (SMCW) chip based on the traditional M-line method^[10]. The M-line spectra were widely applied to the prism-coupled planar waveguide structure for its high sensitivity. It can provide a dramatic view of the mode spectrum by direct visualization of the scattered light, and the observed dark line corresponding to the $m$th mode is referred to as the ‘M-line’. It is not mentioned a lot nowadays, though it is still widely applied in optical sensing applications^[11,12].

The SMCW structure can support various unique oscillating guided modes, i.e., the ultrahigh-order modes (UOMs). Critical coupling of a specific UOM can be achieved when the mode leakage interferes destructively with the reflected beam, resulting in a resonant dip in the reflection spectrum and an M-line in the scattered beams. The high sensitivity of the UOMs is very useful and has been applied in various fields, e.g.,  we have recently demonstrated an enhanced Imbert–Fedorov effect directly without quantum weak measurement^[13,14]. Along with the high mode density and the narrow full-width at half-maximum (FWHM) of the resonant dips, the coupling of the UOMs provides extremely high finesse and high angular resolution to facilitate phase structure detection.

Our experiments show that the reflected pattern varies significantly near the UOMs’ resonance, whilst several concentric rings are severely distorted as the twisted M-line sweeps across the light spot. The usual transfer matrix method for the waveguide fails on this issue since the beam axis of the VB cannot be applied to define the incident angle. Instead of applying Fourier analysis, which describes the light propagation using a plane-wave basis, we find that a simple model in the ray optics picture is sufficient. We replace the common incident angle by a local incident angle, which is defined via the energy flux over the whole cross section of the VB; then, the reflected pattern is simulated with high accuracy. Furthermore, the evolution of the M-line depends crucially on the phase structure and can be used for detection purposes. This work helps us to understand the mode coupling process using twisted photons and may provide an effective approach to the sensing of complicated phase structures.

The schematic in Fig. 1 shows an incident VB reflected by an SMCW chip, and the insert shows the distribution of a specific UOM. The SMCW structure is similar to the metal–dielectric–metal (MDM) scheme that is widely studied in plasmonics. Instead of the basic symmetric and antisymmetric modes, our interest mainly focuses on the oscillating modes of high order. In our experiment, a 0.23 mm thick glass slab with excellent optical parallelism is sandwiched between two silver films. The excited UOMs will propagate along the millimeter thick glass layer until all of the energy is leaked out through the metal claddings. The upper coupling layer is made of 30 nm thick silver film, which facilitates the direct coupling into the chip from free space. No high index prism or coupling grating is needed, since the effective index of UOMs can approach zero. The metal substrate is also 30 nm thick, so both reflection and transmission can be detected. Theoretically, it is simple to calculate the reflectivity and field distribution of a Gaussian beam via the transfer matrix method. What we are more interested in is the reflection of an incident beam carrying OAM at the resonance of the UOMs.

A VB generated by a spiral phase plate is incident on the surface of the SMCW with an angle $\theta$, where the topological charge is $l=1$, and the wavelength and the beam waist are 532 nm and 0.5 mm, respectively. The SMCW chip is placed on a goniometer to carry out angular scanning. The reflection spectrum is recorded by a photoelectric detector, and the reflected pattern is detected by a charge-coupled device (CCD). The typical reflected patterns measured around a UOM’s resonance are illustrated in Fig. 2, whilst the corresponding reflectivity is shown in Fig. 3. Note that the angle corresponding to Fig. 2(c) is located close to the center of a resonant dip, and the reflected spots are significantly distorted in comparison with the incident beam. Careful examination shows that several outer rings other than the inner ring exist in the reflected spot, which evolve continuously as we scan the incident angle gradually. A twisted M-line appears from one side, moves across the light spot, and disappears on the other side. As the twisted dark line passes through the beam center, the reflectivity decreases to a minimum, leading to the resonant dip.

The inner ring is formed by the direct reflection of incidence at the chip surface. The physics behind the occurrence of outer rings can be attributed to the random scattering caused by the natural roughness of the silver film, which enables the coupling between different UOMs of different orders in all azimuthal directions^[15]. Thus, hollow light cones other than the reflected spot are formed due to the leakage of the UOMs. The outer rings in Figs. 2(c) and 2(d) are brighter than that in Fig. 2(b), while the inner ring shows an opposite manner. The reason can be explained as follows: critical coupling of the UOMs can be achieved when destructive interference between the leakage of the UOMs and the direct reflection at the upper surface is complete, and thus the inner ring at resonance is dim. On the other hand, most energy is shifted to UOMs at resonance, so the mode leakage is enhanced to produce brighter outer rings. Given the origins of the inner and outer rings, the rest of this paper will focus on the physics behind the twisted M-line.

In Fig. 3, the measured reflectivity of a VB is compared with the simulated results. Since the number of the UOMs in the sub-millimeter thick SMCW is very large, we only show a part of the spectrum for clarity. The FWHM of the dips in the experiment is much wider, which can be attributed to the different incidence conditions and will be explained later. In Fig. 4, we show the reflection of an ordinary Gaussian beam near resonance, which differs significantly from the VB case. Figures 4(b) to 4(d) show that the Gaussian beam is stretched at first, then a curved M-line appears corresponding to the resonant angle, and finally the Goos–Hänchen effect manifests itself as a discernible shift of the beam peak.

First, we demonstrate the high angular sensitivity of the UOMs, which is crucial to sensing the phase pattern of a VB. To clarify the inner relation among the UOMs, the surface plasmon (SP) mode, the symmetric (${\mathrm{TM}}_0$) mode, and the anti-symmetric (${\mathrm{TM}}_1$) mode, the reflection spectra of a prism-coupled SMCW chip are plotted in Fig. 5. Note that the prism is not necessary for the coupling of the UOMs, since they can be excited at a small incidence angle. The prism facilitates the coupling of the SP and the symmetric modes since they can only be excited under the attenuated total reflection (ATR) condition. As the thickness of the guiding layer increases, the ${\mathrm{TM}}_0$ and ${\mathrm{TM}}_1$ modes degenerate into the SP mode, while the remaining TE and TM modes degenerate into the UOMs. For an SMCW chip of millimeter scale, the total number of the UOMs is in the order of ${10}^3$, and the corresponding FWHM of each resonance dip is much narrower than that of the SP resonance. Consequently, higher angular resolution can be obtained via the UOMs.

Let us further consider the finesse $F$ of the chip, which is defined by the ratio of the mode spacing $\mathrm{\Delta }\theta$ to the mode width $w_{\theta }$. From the dispersion relation of UOMs, it is easy to find that the mode spacing can be approximated as $\mathrm{\Delta }\theta \propto 1/\sin 2\theta$, and the mode width is $$w_{\theta }=\frac{2[\mathrm{Im}({\beta }_{\mathrm{rad}})]}{k{n}_g\cos \theta },$$(1)where $k$ is the free space wavenumber, $n_g$ is the refractive index of the guiding layer, and the radiative damping is^[16]$$\mathrm{Im}({\beta }_{\mathrm{rad}})\approx \frac{\exp (2i\alpha s)}{d\sin \theta }.$$(2)$s$, $d$ are the thickness of the coupling and guiding layers, respectively, and $\alpha$ is the attenuation coefficient of the metal cladding. Substituting Eqs. (1) and (2), the finesse of the UOMs can be written as $$F\propto \frac{n_{g}kd}{\exp (2i\alpha s)}.$$(3)

Under weak coupling condition, there is $\exp (2i\alpha s)\ll 1$. For the millimeter thick waveguide, there is $kd\sim {10}^3$. Thus, the finesse of the proposed SMCW chip is very high, whilst a typical Fabry–Perot cavity only has finesse in the order of ${10}^2$. Using the same parameters as we have applied for the numerical simulation in Fig. 3, the finesse can be evaluated as $3.804\times {10}^3$. The high finesse of our resonant chip can also be illustrated by the extremely narrow dips in the numerical simulation based on planar incidence in Fig. 3. Consequently, the UOMs can provide high angular resolution to facilitate the phase structure sensing of a VB.

Next, we consider a Laguerre–Gaussian beam carrying an optical vortex with a topological charge $l$, whose phase $\mathit{\Phi }$ is given by^[17]$$\mathit{\Phi }=\frac{k{\rho }^{2}z}{2(z^2+z_R^2)}+l\phi +kz-(2p+|l|+1)\arctan (z/z_R).$$(4)

For the parameters, $z_R=k{w}_0^2/2$ is the Rayleigh range, $w_0$ is the beam waist, and $p$ is the radial index. The last term in Eq. (4) is the Gouy phase. Let us assume that the Poynting vector $\stackrel{\rightharpoonup }{S}$ is always perpendicular to the helicoidal phase plane, which follows $$\stackrel{\rightharpoonup }{S}\propto \frac{\partial \mathit{\Phi }}{\partial \rho }{e}_{\rho }+\frac{\partial \mathit{\Phi }}{\rho \partial \phi }{e}_{\phi }+\frac{\partial \mathit{\Phi }}{\partial z}{e}_z.$$(5)

Inserting Eq. (4) into Eq. (5) and neglecting the Gouy phase, we have $$\stackrel{\rightharpoonup }{S}/k\propto \frac{\rho z}{z^2+z_R^2}{e}_{\rho }+\frac{l}{k\rho }{e}_{\phi }+e_z.$$(6)

As shown in Fig. 6(a), it is simple to plot the distribution of $\stackrel{\rightharpoonup }{S}$ on different cross sections of the beam as the light propagates. Obviously, the conventional incident angle $\theta$ defined by the beam axis cannot represent the actual incident angle at specific positions. Thus, when a VB is refracted by an SMCW chip with high angular sensitivity, great care is required.

When the beam is incident with a nonzero $\theta$, the cross section of incidence is also tilted by $\theta$. An example is shown in Fig. 6(b), where a VB travels along the $z$ axis, and the vector $\stackrel{⌢}{n}$ denotes the normal to the chip surface of the SMCW. Apparently, the local incident angle ${\theta }_L$ at a specific point should be defined as the angle between the normal $\stackrel{⌢}{n}$ and the energy flux vector $\stackrel{\rightharpoonup }{S}$ for each point.

For the calculation of the local incident angle ${\theta }_L$ in the Cartesian coordinates, Eq. (6) can be recast into $\stackrel{\rightharpoonup }{S}\propto S_x{e}_x+S_y{e}_y+S_z{e}_z$, with $$\begin{gathered} S_x/S_z=\frac{\rho z}{z^2+z_R^2}\cos \phi -\frac{l}{k\rho }\sin \phi , \\ {S}_y/S_z=\frac{\rho z}{z^2+z_R^2}\sin \phi +\frac{l}{k\rho }\cos \phi . \end{gathered}$$(7)

In view of Fig. 6(b), the normal to the chip surface $\stackrel{⌢}{n}$ can be defined as $(-\sin \theta ,0,\cos \theta )$. Finally, we can write down the local incident angle ${\theta }_L$ as $${\theta }_L=\arccos \frac{\cos \theta S_z-\sin \theta S_x}{\sqrt{S_x^2+S_y^2+S_z^2}}.$$(8)

Eq. (8) can be applied to calculate the distribution of ${\theta }_L$ over the titled cross section at different propagation distances $z$. Note that the VB has a spread at the local incident angle ${\theta }_L$, which explains the wide FWHM observed for VB incidence, as shown in Fig. 3.

As mentioned in Section 2, the outer rings in Fig. 2 are formed by random scattering of the natural roughness and the coupling between UOMs. The inner ring is formed mainly due to the direct reflection of the VB. Based on the proposed theory, the patterns of the inner ring can be simulated by taking the local incident angle ${\theta }_L$ into account, and the results are shown in Fig. 7.

The similarity between Figs. 2 and 7 is obvious, so the proposed model fits well with the experiments. Figure 7 shows the whole process of how the reflected beam is distorted by a twisted M-line. The M-line emerges from the central phase singularities, moves across the beam, and finally disappears into the singularity again. Since the M-line varies continuously as a function of the incident angle, the M-line method via UOMs provides sufficient sensitivity for the phase structure detection of the light beam carrying OAM.

For large incident angle $\theta$, the density of the UOMs increases, which enables the possibility to excite several UOMs simultaneously via a single beam. Thus, more information can be obtained when more complicated phase structures are detected. An example is shown in Fig. 8, where VBs of different charges are incident with a large angle $\theta =30deg$. For both cases of $l=1$ and $l=2$, the twisted M-lines in the intensity profiles match well with the phase patterns. Furthermore, since the large topological charge corresponds to large phase variation, two UOMs (mode $a$ and mode $b$) are successfully excited in the case of $l=2$, resulting in two twisted M-lines across the beam. In comparison, the number of M-lines in the case of $l=1$ is only one. Note that we should not read too much into the center region of the beam, where the phase is ill defined at the singularity.

In conclusion, we investigate the reflection of a VB by an SMCW chip. Owing to the high angular resolution of the UOMs, the phase structure of the reflected beam is well described by the evolution of twisted M-lines. A very simple model is proposed based on defining the local incident angle, and the theoretical prediction matches well with the experiments. This work provides an effective approach to the imaging and sensing of complicated phase structures.