Fourier transform is used to analyze grazing incidence X-ray reflectivity curve of thin film. It is a powerful analysis method for the characterization of layer thickness, interface roughness, etc. In the Fourier transform analysis, the window function has a very important effect on the transform result. By analyzing the Fourier transform of rectangular window (FFT) and wavelet window Fourier transform (WFFT), it is found that WFFT can effectively suppress the appearance of interferential peak in the FFT auto-correlation function, which is very important for identifying true and false peaks and the adjacent peaks. In addition, the film thickness error varies in aperiodic oscillation with the increasing signal length, and this change is associated with the oscillation of reflectivity curve. The ability to distinguish layer thickness is better when intercepting the reflectivity curve near the valley of Bragg diffraction peak. Ni film and Si/Ti/Ni/Ti multilayer are characterized by FFT and WFFT, which verifies the above conclusions and provides a powerful analysis method for the characterization of unconventional multilayer structures.