As Internet-driven traffic continues to increase dramatically, optical interconnect has been proposed to alleviate the communications bottleneck. The wavelength-division multiplexing (WDM) and polarization-division multiplexing (PDM) techniques, combined with the coherent reception and advanced modulation, have been used to achieve the capacity increment in single-mode optical interconnects. Recently, the mode-division multiplexing (MDM) technique has been emerging since it offers a new degree of freedom to multiplex spatial modes in a few-mode fiber or multimode integrated waveguide [1,2]. Compared with the WDM technique using multiple wavelengths, all the mode channels share the same wavelength for the MDM technique. Therefore, power consumption and cost can be effectively reduced, which is particularly attractive for large-scale photonic integration. Many efforts had been involved to achieve mode multiplexing [3–11], and the maximal mode number so far have reached 11 for single polarization. Despite that, it is still incommensurate with the significant development of the WDM technique with typical dozens of wavelength channels [12,13]. A straightforward method to increase the throughput of a single wavelength carrier is to combine MDM with PDM techniques to realize hybrid multiplexing. For this purpose, the mode or polarization multiplexer/demultiplexer should be specifically redesigned to accommodate the other dimension. One way is coupling the modes with different polarizations into the bus waveguide one by one [14,15]. The subsequent couplers must be very elaborately optimized to avoid polarization coupling. In this scenario, high mode-scalability is not easy to be realized due to severe polarization cross talk. The other way is using polarization-transparent couplers, each followed by a polarization beam splitter (PBS), to multiplex two modes with different polarizations [16]. However, one must design each coupler very elaborately to reduce the inter-modal cross talk. It is still a big challenge to achieve hybrid multiplexing.

Inspired by the cube PBS extensively implemented in free-space optics, we propose a novel integrated mode-transparent PBS (MTPBS) to realize hybrid multiplexing/demultiplexing. It splits the hybrid multiplexed signals (MDM and PDM) into dual-path single-polarization MDM signals, which can be subsequently handled by conventional mode demultiplexers individually to obtain separate data channels. Thus, the mode demultiplexer for each polarization can be optimized independently to decrease inter-modal cross talk, alleviating the design complexity greatly. Conventional PBSs based on lateral evanescent coupling [17–19] are difficult to be redesigned to realize mode-transparent manipulation, due to the severe mode dispersion. By contrast, in the proposed MTPBS, with a waveguide width much greater than wavelength, various modes tend to degenerate and behave a similar feature. In addition, being different from the conventional PBSs based on lateral evanescent coupling, the evanescent coupling in the proposed MTPBS is along the waveguide, enabling the simultaneous coupling for multiple modes. Therefore, mode-transparent operating can be fundamentally realized.

As a proof of concept, the MTPBS supporting a total channel number as high as 13, i.e., seven transverse electric (TE) mode channels and six transverse magnetic (TM) mode channels, is successfully demonstrated. For TE polarization, the insertion losses of TE0−TE6 modes are about 0.7–3.3 dB, and the modal/polarization cross talk is about −13/−11dB. For TM polarization, the insertion losses of TM0−TM5 modes are about 1.7–3 dB, and the modal/polarization cross talk is about −7/−17dB.

Figure 1(a) indicates the schematic of the proposed MTPBS adopting a bi-trench coupler. Fully etched trenches in the T-shape structure are aligned 45° with respect to the input waveguide. The light is split by the polarization state: TE polarization is reflected at 90°, while TM polarization is transmitted straightly. The incident angle is greater than the critical angle of total internal reflection (TIR) for both TE and TM polarizations, forming the TIR mirrors. A portion of input power can pass beyond the trench, provided that the trench is narrow enough [20]. The integrated MTPBS is spired by the cube PBS, which has been available commercially in free-space optics, as shown in Fig. 1(b). A pair of right-angle prisms is placed closely with each other, and a polarization-dependent dielectric coating on the hypotenuse of one prism is utilized to separate the right-angle prisms. An anti-reflection coating consisting of multilayer optical films is cemented onto each face of the cube PBS to achieve ultralow surface reflection. The cube PBS and the MTPBS possess similar geometric construction. The right-angle prism corresponds to the TIR mirror, while the dielectric coating corresponds to the trenches. The cube PBS can split arbitrarily polarized light into two orthogonal, linearly polarized components by using the polarization-dependent dielectric coating. P-polarized light is turned with 90°, while S-polarized light is transmitted straightly. Similarly, for the MTPBS, TE polarization can be reflected, while TM polarization is transmitted straightly.

To investigate the mode-transparent operation, we calculate the effective indices of eigenmodes in the waveguide with different widths, as shown in Fig. 2(a). Here, a silicon-on-insulator (SOI) platform with a 220 nm thick top silicon layer and a 2 μm thick SiO2 buffer layer is adopted. The square-curves and triangle-curves correspond to TE and TM polarizations, respectively. It can be seen clearly that the mode dispersion monotonically decreases with the waveguide width. When the waveguide width is a little greater than the wavelength, the mode dispersion is significant, and the mode features are quite different, especially for the lowest- and highest-order modes. When the waveguide width is much greater than the wavelength, the modes with the same polarization tend to degenerate to the slab mode that propagates in the plane as in free space. Thus, different modes are indistinguishable and behave similar features. Note that the TE and TM polarizations cannot degenerate due to the inherent structural birefringence. Therefore, different modes with the same polarization can be tackled simultaneously, reducing the design and optimization complexity greatly. As a demonstration, we design an MTPBS supporting 13 channels, including seven TE and six TM modes. Here, the width of the bus waveguide is set as 25 μm to ensure low modal dispersion for these selected modes.

The trench coupler, which previously served as a power splitter [21–23], is adopted to split polarizations here. The length of evanescent wave d can be obtained approximately by [24] where k0 is the wavenumber in free space, and neff is the effective refractive index of TE or TM polarization. ncl is the refractive index of the trench, and the θi is the incident angle. For TM polarization, the index q=1. For TE polarization,

The neff of TE or TM polarization is set as 2.847 or 2.054 at 1550 nm, and the θi is chosen as 45°. The lengths of the evanescent wave of TE and TM polarizations are calculated as 0.086 and 2.45 μm, respectively. It can be anticipated that TM polarization has a higher transmission efficiency than TE polarization due to stronger evanescent coupling along the waveguide. Therefore, TE and TM polarizations can be isolated by the trench coupler effectively. Note that the working mechanism of the proposed trench coupler is quite different from lateral evanescent coupling in the conventional PBS, where the strong coupling strength and low mode dispersion cannot be satisfied at the same time. To find an optimal structure to increase the TM transmission efficiency and restrict that of TE polarization, bi-trench and tri-trench couplers, which are inspired by the multilayer optical film structures of the cube PBS, are analyzed in addition to the single-trench coupler. The power transmission efficiency at the through port versus different trench widths for TE0 and TM0 modes is calculated, as shown in Fig. 2(b). Figure 2(c) illustrates the schematics of these three kinds of trench couplers. In the bi-trench and tri-trench couplers, the width of the trench equals to that of the gap between the neighboring trenches.

For the TM0 mode, the power transmission efficiency of the single-trench coupler monotonically decreases with the increase of the trench width, and the efficiency is −0.6dB with a trench width of 200 nm, whereas the efficiency reaches a maximum of −0.06dB with the trench width of 300 nm for the bi-trench coupler. The efficiency of the tri-trench coupler has two extremums of −0.08dB with the trench widths of 200 nm and 400 nm, respectively. For the TE0 mode, the efficiencies of the bi-trench and tri-trench couplers are much smaller compared with the single trench case. The extremum of transmission efficiency for TM polarization results from the constructive interference in the multilayer dielectric structure. We also obtain the efficiency using the transfer matrix theory applied in the multilayer dielectric structure, as the solid curves shown in Fig. 2(b). The calculated results by the transfer matrix theory agree quite well with that by the 3D finite-difference time-domain (3D FDTD) method when the trench width is small. The difference between the two methods gets more significant in the case of a wider trench width, and this can be mainly attributed to the loss induced by pinhole diffraction and Goos–Hänchen shift in the trench coupler, which are neglected by the transfer matrix theory. Although the tri-trench coupler has a high transmission efficiency around the trench widths of 200 and 400 nm, the curve of the bi-trench coupler around the extremum of 300 nm is flatter, corresponding to a larger fabrication robustness. Therefore, the bi-trench coupler with the trench width of 300 nm is adopted.

Figure 2(d) exhibits the simulated power transmission efficiency with different trench widths at the through port, for TE0−TE6 and TM0−TM5 modes, respectively. It can be seen clearly the maximal efficiency is obtained with the trench width of 300 nm for TM0−TM5 modes, and the efficiencies for TE0−TE6 modes are all suppressed to lower than −20dB. The simulated spectra in the wavelength range of 1500−1600 nm for TE and TM polarizations are shown in Figs. 2(e) and 2(f), respectively. In the legend “TM0−TM1−T,” the TM0 mode denotes the input mode, while the TM1 mode denotes the output mode. The letter “T” refers to the through port, while the letter “C” refers to the cross port. For the demonstration convenience, we only display the results for the lowest- and highest-order modes. For the TE0 mode, the insertion loss is about 0.02 dB, and the modal cross talk mainly from the TE1 mode at the cross port is about −32dB. Here, the modal cross talk is defined as the difference between the transmission of the desired mode and that of the undesired modes with the same polarization, while the polarization cross talk is defined as the difference between the transmission of the desired mode and that of the undesired modes with the different polarization. For the TE6 mode, the insertion loss is about 0.1 dB, and the modal cross talk mainly from the TE5 mode is about −22dB. The polarization cross talk from TM0 or TM5 at the cross port is about −18 or −13dB, respectively. For the TM0 mode, the insertion loss is about 0.06 dB, and the modal cross talk mainly from the TM1 mode at the through port is about −24dB. For the TM5 mode, the insertion loss is about 2.5 dB, and the modal cross talk mainly from the TM4 mode is about −10dB. The polarization cross talk from TE0 or TE6 at the through port is about −30 or −25dB, respectively. We can see that the insertion loss is higher for high-order modes. The reason is that the effective index is quite different between high-order and low-order modes, while the currently utilized bi-trench coupler performs better for the low-order modes, and obvious reflection can be found for high-order modes. An effective and simple method is to widen the waveguide so that more modes tend to degenerate. Therefore, a same device geometry can accommodate more modes, and the losses of high-order mode can be reduced effectively.

Figures 3(a) and 3(b) show the light propagation in the bi-trench coupler for TM0 and TM5 modes at 1550 nm, respectively. The cases for TE0 and TE6 modes are illustrated in Figs. 3(c) and 3(d). The TM modes are both transmitted to the through port, while the TE modes are reflected to the cross port. All the modes stay almost unchanged. Weak light scattering for the TM5 mode results from the pinhole diffraction, which can be alleviated by adopting a wider waveguide. The polarization cross talk from TM polarization can be further cut by cascading a same MTPBS at the through port.

The proposed MTPBS was fabricated on an SOI platform with a 220 nm thick top silicon layer and a 2 μm thick SiO2 buffer layer. The waveguide structures were formed by the electron beam lithography process to define the pattern and inductively coupled plasma process to fully etch the silicon core layer. A 1 μm thick SiO2 cladding by the plasma-enhanced chemical vapor deposition process is deposited onto the whole device as the cladding. The fabricated MTPBS is shown in Fig. 4(a). The entire tested device consists of additional TE and TM mode multiplexers/demultiplexers [unshown in Fig. 4(a)] and three adiabatic tapers. To fully characterize the performance for TM and TE inputs, two reference structures with the same MTPBS but different mode multiplexers are fabricated. In addition, reference MDM structures comprising only a single-polarization mode multiplexer and demultiplexer are fabricated on the same chip. A silicon waveguide directly connected by a pair of TE/TM grating couplers is also fabricated as a reference. Figures 4(b) and 4(c) show the seven-TE-mode (de)multiplexer and the six-TM-mode (de)multiplexer to attain the single-polarization mode multiplexed signals individually. The TE/TM mode (de)multiplexers are connected to the MTPBS by the adiabatic taper. Figure 4(d) illustrates the zoom-in view of the MTPBS, where the additional TM grating is designed to filter out the reflected light when TM-polarized light is launched, avoiding twice reflection into the waveguide.

The seven-TE-mode (de)multiplexer consists of six cascaded adiabatic couplers (ACs) to excite TE1−TE6 modes. For the ith AC to achieve TE₀-TE_i mode conversion, two reversely tapered waveguides (access waveguide and bus waveguide) are placed closely to form a coupling region, as shown in Fig. 5(a). The six-TM-mode (de)multiplexer is composed of five cascaded asymmetric directional couplers (ADCs) to excite TM1−TM5 modes. For the ith ADC to achieve TM₀-TM_i mode conversion as shown in Fig. 5(b), the widths of the two straight waveguides (wa and wb) are chosen according to the phase-matching condition for TM polarization. The TM_i mode can be fully excited by adopting a proper coupling length. The detailed parameters of TE and TM mode multiplexers are exhibited in Fig. 5(c). The measured insertion losses for TE0−TE6 modes are about 0–2.5 dB, and the modal cross talk is −14.5 to −12dB in the wavelength range of 1520–1600 nm. The insertion losses for TM0−TM5 modes are about 0–9 dB, and the modal cross talk is about −16 to −15dB in the wavelength range of 1520–1560 nm.

By subtracting the loss caused by the grating couplers, mode multiplexers, and mode demultiplexers, we obtain the normalized response of the MTPBS when injecting TE- and TM-polarized MDM signals, as exhibited in Figs. 6 and 7. In the legend “TE1−C,” TE1 denotes the output mode, while the letter “C” refers to the cross port. In the legend “Cross talk-T,” “Cross talk” refers to the maximal cross talk at the through port which mainly comes from the adjacent modes, and the letter “T” refers to the through port. The insertion losses for TE0−TE6 modes are about 0.7, 0.7, 1.4, 1.8, 1.9, 1.1, and 2.3 dB, respectively, and the modal cross talk is about −12 to −8dB in the wavelength range of 1520–1560 nm. The polarization cross talk from TM polarization is about −11dB, as the purple curves shown in Fig. 7. The insertion losses for TM0−TM5 modes are about 1.7, 5.4, 5.4, 2.0, 1.8, and 3.0 dB, respectively. We can see the insertion losses and the modal cross talk for TM1 and TM2 are larger. It can be attributed to imperfect fabrication of TM0−TM1 and TM0−TM2 mode converters, resulting in that the TM1 and TM2 modes cannot be excited and extracted independently. The modal cross talk for TM0,TM3,TM4, and TM5 is about −9 to −7dB. The polarization cross talk from TE polarization is lower than −17dB, as the purple curves shown in Fig. 6. Note that the modal cross talk of the MTPBS is mainly limited by that of TE and TM mode (de)multiplexers, which can be effectively improved by adopting high-performance mode (de)multiplexers.

In summary, we propose a novel mode-transparent PBS that enables hybrid mode and polarization multiplexing. Spired by cube PBS in free space, we adopt a bi-trench coupler to separate the TE and TM polarizations effectively. By adopting waveguide width much greater than the working wavelength, mode-transparent manipulation and mode scalability can be both solved fundamentally. As a demonstration, the MTPBSs handling seven-TE-mode and six-TM-mode channels are experimentally realized, enabling a hybrid (de)multiplexer with a record-high channel number of 13. The work gives a universal and general solution to combine both MDM and PDM techniques and provides a different solution of large-scale integration for ultrahigh bandwidth communications.