Based on noise characteristics of wireless optical communication, atmospheric channel model is established. Clustering analysis in constellation is used in subcarrier M-ary phase shift keying (MPSK) modulation under different scintillation distribution. While turbulence intensity is wake, MPSK (M≤8) modulation signal could get high recognition rate arriving 88%. With the increasing of turbulence intensity, clustering performance of constellation becomes poor, and the recognition rate of 16PSK decreases. The simulation results show that the method studied here has certain applied prospect in subcarrier modulation signal recognition over wireless optical communication.