S-transform, combining the advantages of both the windowed Fourier transform and the wavelet transform, is suitable for the demodulation of non-stationary signal such as the deformed fringe pattern. The fringe patterns analyzed by S-transform are discrete in application. During the calculation of the S-stransform coefficients, the Fourier spectrum of the signal is translated, which means that the frequency information of the adjacent island is used. The frequency overlapping caused by sampling, nonlinearity or height variation of the measured object affects the three-dimensional reconstruction result. The effects from the above factors in calculating S-transform coefficients are exhaustively studied from the point of view of the frequency analysis. The frequency-domain description and the measurement range of S-transform coefficients of digital signal are deduced when the nonlinearity from both the projector and CCD exists. The sampling condition and the structure condition are given as well. Both computer simulation and experimental results verify the analysis.