The adaptive compensation method of polarization mode compensation (PMD) has been studied in depth. Degree of polarization was sampled as the feedback signal and particle swarm optimization (PSO) algorithm was applied as the feedback control method in the adaptive compensation system. The effect of the compensation system upon signals of different modulation formats was examined and carried into comparison experimentally. The quality of the transmission signal being severely deteriorated, signals of different modulation formats were compensated productively with bit error ratio (BER) improved enough to satisfy the demand of optical communication systems (10-9). Thus the insensibility of the productiveness of PMD adaptive compensation system towards signals of different modulation formats was experimentally proved. Besides, the NRZ possess a certain power penalty over each other modulation format, while the receiving performance of signal of RZ50 was the best.