The monitoring performance of monitor window and degree of polarization (DOP) sensitivity to defferetial group delay (DGD) can be improved greatly by using the band-pass and band-reject filters are presented in the polarization mode dispersion (PMD) compensation system, where the DOP is used for the feed back signals. A comparison is made for the filters in improving the monitoring performance of the feed back signals between the 40 Gbit/s return-to-zero (RZ) and 10 Gbit/s nonretrun-to-zero (NRZ) transmission system, and numerical analysis is made for the selection of the bandwidth of the filters. The results showed that the local extremum can be avoid in the searching process of the searching algorithm, so a bypass searching is reduced greatly, which is significant for the compensation effects and the stabilization of the system.