To solve the coupled wave function of nonlinear recording hologram, the linearization method was advanced, and two modulating models of refractive index were adopted, the convex curve and the concave curve. Taking the reflective phase hologram grating as an example, the spectral properties of the gratings were analyzed, for recording material with refractivity modulated in the convex curve and concave curve, by the coupled-wave theory. There is great influence on the bandwidth of the spectral properties from different modulating models of refractive index, and it affects the secondary peak value, but the primary peak value and position. The results can explain the diversity of spectral property and the experimental bandwidth wider than that calculated.