Existing methods for detecting the number of vehicle occupants in a high-occupancy vehicle (HOV) lane, using radar and infrared thermal imaging technology, exhibit low reliability and low accuracy. To address these limitations, a method for detecting the number of vehicle occupants based on multispectral infrared imaging and an improved Faster regions with convolutional neural networks (R-CNN) algorithm is proposed. The vehicle interior space image is obtained using a multispectral infrared imaging system, and the number of passengers is detected by a Faster R-CNN deep learning algorithm. The generalization ability of the network is enhanced using the full convolution network structure and multiscale feature prediction, and ROI-Align is used instead of ROI-Pooling. Through K-means clustering, the prior distribution of the geometric proportion of the length and width of the target frame is obtained, which improves the training speed and the accuracy of position regression of the region proposal network (RPN). The test results showed that the interior space image was clear, and the algorithm could detect the number of passengers. After its improvement, the generalization ability of the network was enhanced, and the accuracy of single occupant detection reached 88.6%, which was 13.8% higher than before its improvement. This meets the requirements of more than 80% of industry regulations.