Based on the actual interferogram grabbed by a ZYGO interferometer, the phase is retrieved by two-dimensional fast Fourier transform (FFT) method and the factors which impact phase retrieval accuracy of FFT method including edge errors, window function, filter design, interferogram extrapolation and the number of carrier fringes are comprehensively analyzed, by comparing with the phase-shift measurement results. It is found that for the direct 2D FFT method, the edge 0.05R circular area of the interferogram has the biggest phase retrieval errors and decides the value of global errors. Window functions almost have no improvement for edge errors. Optimization of filter design can improve edge errors in some degree. Interferogram extrapolation can reduce edge errors effectively and it is the best way to enhance phase retrieval accuracy for continuous optical surface test. Accurate measurement results can be achieved when the frequency is 1/13~1/3 of the space resolution of the interferogram with peak-to-walley (PV) value of phase retrieval errors less than λ/20. The higher the carrier frequency is, the better the detail resolution is.