A homodyne coherent laser communication system has the characteristics of high speed and long-distance transmission, and it plays an important role in the establishment of inter-satellite high-speed laser communication network. The operating principle of optical phase-locked loop is analyzed, and the corresponding mathematical model is established. The open-loop and closed-loop transfer functions of optical phase-locked loop are obtained. The error transfer function is derived. The influencing factors on loop bandwidth are analyzed. The phase noise error of phase-locked loop is analyzed， and the relationship among error factor, detection sensitivity and loop bandwidth is established. The optimized bandwidth of phase-locked loop is 1.5 MHz, and the bit error rate of the receiver is tested. The receiving sensitivity of -41.4 dBm and the bit error rate of 10-7 are obtained when the communication rate is 5 Gbit/s and the modulation mode is binary phase shift keying.