In the coherent laser communication systems, the instrumental polarization aberration of the optical system can change the polarization state of the received signals and reduce the heterodyne efficiency between the local oscillator beams and received signals. In order to quantitatively analyze the polarization properties of the optical system, the three-dimensional polarization ray-tracing algorithm is proposed. Based on a large aperture coherent laser communication testing platform system, the polarization ray tracings of the communication channel are performed by means of this algorithm, and the influences of the polarization aberration on the polarization state of the circular polarization signal are discussed according to the calculated polarization transformation matrices. The polarization characteristics of the communication system of the testing platform are verified through the experimental platform.