Mode- and polarization-division multiplexing are new promising options to increase the transmission capacity of optical communications. On-chip silicon polarization and mode handling devices are key components in integrated mode- and polarization-division multiplexed photonic circuits. In this paper, we review our recent progresses on silicon-based polarization beam splitters, polarization splitters and rotators, mode (de) multiplexers, and mode and polarization selective switches. Silicon polarization beam splitters and rotators are demonstrated with high extinction ratio, compact footprint and high fabrication tolerance. For on-chip mode multiplexing, we introduce a low loss and fabrication tolerant three-mode (de)multiplexer employing sub-wavelength grating structure. In analogy to a conventional wavelength selective switch in wavelength-division multiplexing, we demonstrate a selective switch that can route mode- and polarization-multiplexed signals.