The transmission and reflection spectra of the rough thin films are directly affected by the surface roughness. If this effect is not taken into account, there will be evident error for the calculation of thickness and optical constants of rough thin films. Using the scalar wave scattering theory, the light scattering caused by rough film surface is analyzed in detail by introducing the surface root mean square roughness, and the transmittance expression of the thin film system is obtained. On this basis, the calculated film thickness and transmission spectrum are in agreement with the measurement results of the prepared hydrogenated amorphous silicon (a-SiH) thin film. The determined optical constants are closer to the actual ones. Because the calculation process is not based on the minimum value optimization algorithm and does not need the aid of complicated software, it is an effective and exact method to determine the thickness and optical constants of rough thin films.