Chip-scale multi-dimensional multiplexing technology that combines wavelengths and spatial modes on a silicon photonic integrated circuit (PIC) is highly promising for the link-capacity scaling of future optical interconnects. However, current multi-dimensional multiplexed PICs face significant challenges in simultaneously achieving broad optical bandwidth, low mode crosstalk, and dual-polarization modes in an ultra-compact footprint as the number of spatial modes increases. To address the issue, a topology-optimization-based inverse design assisted by a novel manufacturing calibration method (MCM) is utilized. Based on a 220 nm silicon-on-insulator (SOI) platform, a 100 nm broadband and ultra-compact ($6\mathrm{\mu m}\times 6\mathrm{\mu m}$) multi-dimensional multiplexed PIC supporting ${\mathrm{TE}}_0$, ${\mathrm{TE}}_1$, ${\mathrm{TM}}_0$, and ${\mathrm{TM}}_1$ modes with modes crosstalk $<-16\mathrm{dB}$ ranging from 1500 to 1600 nm is demonstrated for the first time, to the best of our knowledge. Furthermore, the PIC is implemented to experimentally enable a single-wavelength 4-modes $\times 100\text{Gbit/s}$ PAM-4 direct modulation data transmission over 51 wavelengths with 0.8 nm channel spacing. This work shows the potential of utilizing multi-dimensional multiplexed PICs as optical interconnects to effectively address the speed limits of data transfer for future high-performance chip-to-chip interconnection.