For one-dimensional semitransparent medium layer with rough surface and Gaussian distribution, the transfer probability model of the spectral radiation on rough surface with shading effect considered is established with microfacet slope method. Monte Carlo method is used to simulate the radiation transfer. The reflection and refraction of spectral radiation at rough surface, absorption of medium and specular reflection on the substrate are considered along with the effects of shadowing and masking. The effects of roughness, optical thickness, refractive index and reflectivity of substrate on bi-directional reflectance distribution function (BRDF) are investigated. The results show that the change trends of reflection peak with incident angles for different rough levels of surface are different, the diffuse component increases with roughness and refractive index, and reflectivity of substrate and optical thickness change the value mainly instead of the distribution trend of BRDF.