A method for calculating diffraction efficiencies of non-ideal volume phase holographic gratings is presented by introducing a function between modulation profiles and depths of gratings. By applying Kamiya rigorous delamination-calculation method, rigorous coupled-wave equation for the non-ideal conditions, such as mixed modulation and attenuated modulation profiles, is expressed. The diffraction characteristics of the phase gratings, mixed gratings, out-of-phase gratings and attenuated gratings are investigated. The response degree of diffraction efficiency to the numbers of sub-regions that have been divided is discussed. Comparing with Kogelnik theory and Sabol model, the accuracy of the algorithm is further demonstrated. The results show that, the peaks of diffraction efficiency change over average absorption constants, attenuated constants and the spatial dephasing between the absorption and phase gratings; absorption modulation causes a modest increase of the peak, but average absorption constants drag the efficiencies of the whole transmission orders. The effect of the numbers of sub-regions that have been divided on the angle selectivity and angular spectrum widths of the grating is small, but it decides the accuracy of the algorithm. The work have a certain reference value for calculating the efficiencies and analyzing the diffraction characteristics of the non-ideal modulation of volume holographic gratings.