In order to understand the static birefringence and the circular dichroism of a photo-elastic modulator (PEM), an intensive study about the polarization characteristics of PEM is done. First of all, Muller matrices are used to describe the polarization characteristics of the photo-elastic modulation, the static birefringence and the circular dichroism. The general polarization modulation model of PEM is built. Then, the principle of the modulation model is studied, theory and simulation are analyzed. Finally, experimental apparatus is established and the polarization modulation experiment is carried out. The results show that the theory model built is reasonable and correct, the polarization modulation characteristic of PEM should be the combination of the small static birefringence,the photo-elastic modulation and the circular dichroism. Moreover, preliminary measurement for one example of PEMs shows that the circular dichroism is 2.755 × 10^-2 , the phase retardation of static birefrigence is b = 5.73 × 10^-2 rad and its direction angle is α = 3.96° . This study gives the general Mueller matrix form of a photo-elastic modulator, and expands the knowledge of the photo-elastic polarization modulation.