In order to quantitatively discuss the influence of atmospheric refraction on radiative transfer process, a vector radiative transfer model (VSPART) is established and the atmospheric refraction impact on diffuse light’s Stokes vectors and radiative flux density is discussed. In this model, the atmospheric refraction parametric process is simulated by using the ray-tracing method, and the radiative transfer equation is solved by using the matrix algorithm. The simulated results of VSPART agrees with literature results and classical radiative models such as SPDISORT(spherical discrete ordinate method), DISORT(discrete ordinate method), RT3/PolRadtran and MYSTIC (Monte Carlo code for the physically correct tracing of photons in cloudy atmospheres). The impact of atmospheric refraction on the Stokes vectors of down-welling diffuse light at surface (DLS) and up-welling diffuse light at the top of atmosphere (ULT) is analyzed for Rayleigh scattering atmosphere and atmosphere with aerosol. Besides, the variation of the atmospheric refraction effect with the solar zenith angle is analyzed as well. Simulation results show that the relative deviations of the diffuse light’s I, Q and U components due to refraction achieve 9.2%, 10.2% and 11.3%, and the relative deviation of radiant flux density approaches 5.3% when the solar zenith angle is 86° in the Rayleigh scattering atmosphere. For DLS, the impact of refraction is enhanced with the increasing of observing zenith angle, while for ULT, the fact is opposite. On the whole, the influence of atmospheric refraction on DLS is stronger than that on ULT. With the increase of aerosol optical thickness, the atmospheric refraction effect is gradually enhanced, and the influence of the atmospheric refraction for soot aerosol is much stronger than that for mineral dust and sea salt. When the solar zenith angle is larger than 70°, the impact of atmospheric refraction increases rapidly, and it is necessary to consider the atmospheric refraction process when simulating the radiative transfer.