Banknote which is issued by the government and forced to use. In 2019, the People’s Bank of China issues the fifth set of RMB. The anti-fouling protection coating is used on both sides, which significantly improves the cleanliness of paper currency. In order to control the coating quality more reasonably, it is necessary to detect the coating thickness in the production. The coating thickness of this paper was to investigate, this paper establishes the optical diffuse reflection model of coating thickness with banknote pattern as a complex substrate surface, identifies and quantitatively detects the coated and uncoated banknote by using Fourier near-infrared spectrometer and confocal laser scanning microscopy system. In this paper, one analytical method based on the combination of multivariate scattering correction (MSC) and second-order derivative combination analysis is proposed for the near infrared (NIR) absorption spectrum data of the coating, that could be effectively identified in the NIR spectrum, and 4 346.764 cm^-1 is determined as the characteristic wave number. Then, the coating thickness model is decoupled basing on the reflectance and roughness. Finally, the coating thickness changes of coated banknotes are detected by a confocal laser scanning microscopy system, and they are correlated with the decoupling result of the model to obtain the actual coating thickness. The final results show that the minimum is 3.807 μm, the maximum is 12.738 μm. The detection method has an important practical significance for coating quality control in banknote production.