At present, infrared image data of photovoltaic modules obtained by unmanned aerial vehicle is more and more used in the fault detection of photovoltaic module. However, due to the high similarity of various samples of photovoltaic module infrared image data, the existing deep learning model has a low ability to extract photovoltaic module infrared image features, resulting in low detection accuracy of photovoltaic module multi-fault types. To solve the above problems, a residual photovoltaic network (ResPNet) model is constructed based on the residual network (ResNet) model for infrared image fault detection of photovoltaic modules. On the basis of ResNet model, ResPNet adds the underlying feature information enhancement module, multi-scale feature information enhancement module and global feature information enhancement module to improve the infrared image feature extraction ability of photovoltaic modules. Experiments are conducted on Infrared Solar Modules, a disclosed infrared image dataset of photovoltaic modules. The ResPNet model achieves an infrared image classification accuracy of 84.6% for 12 types of photovoltaic modules, which is better than not only ResNet-50 model, but also other infrared image classification models. Through cascading several ResPNet models, the highest known infrared image classification detection accuracy of 12 types of photovoltaic modules in this dataset is achieved at 85.9%.