Diffraction characteristics of reflection volume holographic gratings at nonuniform swelling or shrinking by introducing a function between swelling factor and the thickness of the grating layer are investigated. By means of Kamiya′s rigorous delamination-calculation method and Lorentz-Lorenz formula, the analytical expressions of the average refractive index and the modulation are derived. Accordingly, the diffraction efficiency curves of the 1st-order reflection are given when the swelling factors are constant, linear function and nonlinear function. The influences of the average refractive index and the modulation for an ideal grating or decay distribution for the nonideal grating on the angular and wavelength selectivity are analyzed. The diffraction characteristics of the grating are given when the swelling factors are unequal at the exposure area and the unexposed area, and the response degree of diffraction efficiency of the grating to the change of swelling factor is discussed. The results show that nonuniform distribution of the swelling factor causes a diffraction efficiency curve in the asymmetric profile; compared with the modulation of refractive index, the Bragg angle of grating is highly sensitive to the change of average refractive index; the swelling factors at the exposure area and the unexposed area affect the peak efficiency, the angular band range and the spectral bandwidth. The results have theoretical guiding sense for the swelling and shrinking process of the reflection volume holographic gratings.