Coherent laser communication has the advantages of high speed and long-distance transmission capabilities, both of which play an important role in inter-satellite communication. A coherent optical communication system makes full use of information on light intensity, phase, frequency, and polarization, which cannot only improve the efficiency of frequency band utilization, but also extend the relay distance across which optical communication is possible. As a type of multi band modulation, quadrature phase shift keying (QPSK) modulation and demodulation can greatly improve the utilization of spectrum resources. Aiming to resolve the problem of the response rate of QPSK modulation and demodulation in coherent optical laser communication, an improved Costas loop is proposed in this study. The results of the experiment verify the entire QPSK modulation and demodulation system and the performance feasibility of the improved Costas loop.