The effect of the surface albedo on the sky polarization mode is studied. At present, there is a lack of research on quantitative tests with a single influencing factor at different time and locations in the actual environment. In view of this, the re-developed libRadtran software is used to simulate and analyze the sky polarization mode at different surface albedos. Then, synchronous remote tests are conducted in group on meadow and asphalt surfaces, ocean and asphalt surfaces with the imaging test system and the influence of surface albedo on the polarization azimuth and polarization degree is analyzed quantitatively. The results show that the surface albedo has little effect on the polarization azimuth distribution, which is always stable. However, it has a strong attenuating effect on the polarization degree. At the same time on the same day, a greater surface albedo entails a smaller maximum polarization degree; a bigger difference of surface albedo means a sharper average decrease and a larger decreasing extent in maximum polarization degree.