S-Transform, a hybrid and extension of the short-time (or windowed) Fourier transform and the wavelet transform, is one of lossless and reversible time-frequency analysis methods, which is suitable to analyze non-stationary signals. It not only has advantages of linearity, multi-resolution and uniqueness of inverse, but also its inverse transform directly keeps in contact with the Fourier transform. In S-Transform, the harmonic wave is used as a basic element function, and the window function is a Gaussian function with the ability of both dilation and translation, which is controlled by a frequency parameter. Compared with the short-time (or windowed) Fourier transform, it has optimized the time-resolution and frequency resolution simultraneously. Compared with wavelet transform, it keeps in contact with the Fourier transform. S transform in the application of demodulation of fringe pattern with nonlinear parts has been deeply discussed, S-transform expression of the deformed fringe pattern considering by nonlinear effects is deduced and two ways, including S transform filtering method and S transform ridge method are proposed, which are used to eliminate the nonlinear error in three-dimensional optical measurement based on the structured light projection. Computer simulations and experiments have verified the proposed two methods. Compared with Fourier transform and wavelet transform, the proposed methods based on S transform have better reconstruction results.