Free space optical communication(FSO) has the advantages of high bandwidth, no need to lay fiber, low power loss, so it has attracted attention. In Gamma-Gamma atmospheric turbulence channel model, weak, medium and strong three atmospheric turbulence intensity were respectively considered, and combined with the phase noise, the performance of circular polarization modulation system with coherent detection was studied, and the closed form expression of the bit error rate(BER) and outage probability were derived. In the different atmospheric turbulence intensity and phase noise, the performance of the circular polarization modulation system with the coherent detection was simulated and analyzed. Compared with the error performance of the direct detection system, the results show that the error performance of the coherent detection system is better than that of the direct detection system. Therefore, in the circular polarization modulation system, employing the coherent detection method can reduce the effect of the atmospheric turbulence and greatly reduce the BER, and improve the performance of the communication system.