Ta2O5 and SiO2 single-layer films and multilayer high reflectance films have been prepared by reactive magnetron sputtering. Influences of the process parameters on the quality of optical films and the underlying mechanisms were studied. A passive control method based on fitting the optical spectra can effectively improve the quality of the multilayer films prepared in the transition region. Results indicate that in addition to the high deposition rate, films deposited in the transition region also have higher refractive index and lower optical losses, compared with those deposited in the oxide region. The changes of the deposition rates during the deposition process of a multilayer film in the transition region depend on the changes of the sputtering voltage, which can be compensated with the fitting method by monitoring the sputtering voltages. 40-layer Ta2O5/SiO2 high reflectance films were prepared in the transition region on fused silica substrate, which has a surface roughness of 0.56 nm. The reflectance of these mirrors reaches 99.96% as measured by cavity ring-down spectroscopy.