An advanced serial minimum frequency-shift keying (SMSK) modulation scheme is proposed. SMSK signal has a narrow power spectrum, and continuous phase shift that changes π/2 twice during one symbol time, which is similar to traditional minimum frequency-shifts (MSK) signal. The principle of generating the optical SMSK signal and its implementation in octal multi-dimensional and multi-level modulation scheme is presented. Theoretical analysis is given to modulated spectra, residual dispersion tolerance, self-phase modulation (SPM) tolerance and effect of filter bandwidth on it. 120 Gb/s simulated transmission over 350 km fiber link with dispersion compensation is demonstrated.