To improve the accuracy of reference phase estimation in the continuous-variable quantum key distribution, a scheme of reference phase estimation is proposed, which is based on the least square method. Utilizing the second-order correlation characteristics of slow phase drift and the least-square estimation theory, the phase measurement results for a group of reference pulses are fitted by a quadratic polynomial at the receiver to suppress the effect of phase noise. The simulation results show that the least square phase estimation algorithm, compared with the block-averaging estimation algorithm, is better adapted to phase drift variation, so that the mean square error of reference phase estimation can be reduced and the secret key rate of the practical system can be improved significantly.