The propagation model for the spectral combining system of fiber laser beams by the volume Bragg grating (VBG) in the photo-thermal refractive (PTR) glass is built. Considering the surface distortion of the VBG by high-power lasers, the VBG is dealt with by splitting into slice and the wave equation is calculated iteratively by using the finite-different Grank-Nicholson and sparse matrix methods. The intensity distribution, the combination efficiency and the power in the bucket of two fiber laser beams after spectral beam combination by the VBG with different deformations are simulated numerically. The thermal deformation effect of the VBG on the intensity distributions, the combination efficiency and the power in the bucket of the combined beam is analyzed quantitatively. The results show that the surface distortion of the VBG due to high-power lasers not only degrades the beam quality of the combined beam, but also decreases the combination efficiency of the spectral beam combining system dramatically.