The refractive-index distribution function of long-period fiber grating (LPG), which is ultraviolet-written by amplitude mask, is a rectangular wave in the core region. The theoretical method is presented to analyze the characteristics of LPG with the rectangular-index modulation. Here, the weakly guiding scalar approximation and coupled-mode theory based on the three-layer step-index optical fiber are used. The approximate numerical method to deal with the coupled-mode equations is presented in detail and the rationality of the method is given. The transmission spectra of LPG are numerically calculated by using the mathematical software, Matlab. It's found that the transmission spectra of LPG with the rectangular-index modulation are not the linear superposition of its cosine gratings' spectra. The effect of the structural parameters of gratings, such as ambient refraction index, the radii of fiber cladding and the duty cycle of fiber grating, on the transmission spectra of LPG with the rectangular-index modulation is analyzed. Furthermore, the response of double-resonant peak position of a single cladding mode to the parameters of gratings is studied.