On-chip polarization controllers are extremely important for various optical systems. In this paper, a compact and robust silicon-based on-chip polarization controller is proposed and demonstrated by integrating a special polarization converter and phase shifters. The special polarization converter consists of a 1×1 Mach–Zehnder interferometer with two polarization-dependent mode converters at the input/output ends. When light with an arbitrary state of polarization (SOP) is launched into the chip, the TE0 and TM0 modes are simultaneously excited. The polarization extinction ratio (PER) and the phase difference for the TE0/TM0 modes are tuned by controlling the first phase shifter, the polarization converter, and the second phase shifter. As a result, one can reconstruct the light SOP at the output port. The fabricated polarization controller, as compact as ∼150μm×700μm, exhibits an excess loss of less than 1 dB and a record PER range of >54dB for arbitrary input light beams in the wavelength range of 1530–1620 nm.