There are noticeable errors at the border of the pattern when the traditional Fourier transform method (FTM) is used to evaluate the carrier intefergram fringe pattern. A new extrapolation of fringe prolongation method for interferogram fringe pattern analysis based on FTM is proposed to eliminate boundary effects. This extrapolation algorithm is derived mathematically. And numerical simulations are performed to evaluate the performance of the method for one-dimensional digital signal and two-dimensional spatial carrier fringe pattern. The source of error is analyzed, and a comprehensive comparison is made between results from traditional FTM and this new extrapolation. The results show, the rather large error caused by the boundary effect in the course of carrier interferogram fringe pattern treatment with traditional FTM, is reduced effectively. In the wave surface measurement based on spatial phase modulation, the calculating precision for phase reaches 3.3 mrad by treating the spatial carrier fringe pattern with the new method.