Optical elements are the core components of laser systems, and their health status is the key to the stable operation of laser systems. How to realize real-time monitoring and fault diagnosis of optical elements in the working status of laser systems is a problem that urgently needs to be solved in this professional field. To solve this problem, this paper proposes a fault diagnosis method for optical elements based on infrared and visible light videos. Firstly, a long-wave infrared camera and a visible light camera are used to collect video information during the working process of the optical element. Then, the collected video information is processed using anomaly point detection algorithms. Finally, the fault diagnosis and localization of the optical element are carried out in combination with the thermal rise characteristics of the optical element. The experimental results show that, under the same algorithm, the method proposed in this paper has improved the fault diagnosis precision rate, false alarm rate and missed alarm rate by 9.70%, 3.60% and 6.10%, respectively, compared with the method of fault diagnosis using infrared videos alone; the method proposed in this paper has improved the fault diagnosis precision rate, false alarm rate and missed alarm rate by 18.00%, 16.00% and 2.00%, respectively, compared with the method of fault diagnosis using visible light videos alone.