The optical 8-DPSK is a new modulation format. It has narrower spectrum structure, higher spectrum efficiency, nearly invariable envelope, and better nonlinearity suppression characteristic. According to the principle of 8-DPSK, the modulation and demodulation in the back-to-back system of 8-DPSK is analyzed via simulation. Moreover, the polarization-mode dispersion compensation performance of 8-DPSK is studied. The relationship between the signal’s degree of polarization (DOP) and polarization-mode dispersion (PMD) of different duty-cycle signals is analyzed theoretically and numerically. The result shows that the DOP sensitivity is different among different duty-cycle signals, because the change in duty-cycle results in the change in spectrum structure. In the polarization-mode dispersion compensation system, DOP is adopted as the feedback signal, therefore, the performance of compensation is different among different duty-cycle signals. Through simulation analysis, CSRZ-8-DPSK has the best compensation performance thanks to its narrower spectrum structure and carrier-suppressed characteristic.