Microring-based optical switches are promising for wavelength-selective switching with the merits of compact size and low power consumption. However, the large insertion loss, the high fabrication, and the temperature sensitivity hinder the scalability of silicon microring optical switch fabrics. In this paper, we utilize a three-dimensional (3D) microring-based optical switch element (SE) on a multi-layer ${\mathrm{Si}}_3{\mathrm{N}}_4$-on-SOI platform to realize high-performance large-scale optical switch fabrics. The 3D microring-based SE consists of a $\mathrm{Si}/{\mathrm{Si}}_3{\mathrm{N}}_4$ waveguide overpass crossing in the bottom and the top layers, and ${\mathrm{Si}}_3{\mathrm{N}}_4$ dual-coupled microring resonators (MRRs) in the middle layer. The switch is calibration-free and has low insertion loss. With the 3D microring-based SEs, we implement an $8\times 8$ crossbar optical switch fabric. As the resonance wavelengths of all SEs are well aligned, only one SE needs to be turned on in each routing path, which greatly reduces the complexity of the switch control. The optical transmission spectra show a box-like shape, with a passband width of $\sim 69\mathrm{GHz}$ and an average on-state loss of $\sim 0.37\mathrm{dB}$. The chip has a record-low on-chip insertion loss of 0.52–2.66 dB. We also implement a non-duplicate polarization-diversity optical switch by using the bidirectional transmission characteristics of the crossbar architecture, which is highly favorable for practical applications. 100 Gb/s dual-polarization quadrature-phase-shift-keying (DP-QPSK) signal is transmitted through the switch without significant degradation. To the best of our knowledge, this is the first time that 3D MRRs have been used to build highly scalable polarization-diversity optical switch fabrics.