This paper presents a review on the recent progress of digital signal processing (DSP) in the high-speed optical transmission system using single-carrier based multi-level modulation formats. The principle of several digital phase and polarization tracking algorithms recently proposed and demonstrated for future spectrally-efficient optical transmission system are discussed in detail. A novel DSP-based interference mitigation algorithm for the single-ended coherent receiver has been included. Recent technology advance on transmitter-side DSP such as digital pulse-shaping, pre-equalization and digital nonlinear compensation has also been discussed.