In view of the low detection accuracy and high complexity of current multi-scale vehicle detection algorithms in infrared scenes, an infrared vehicle detection algorithm based on Shuffle-RetinaNet is proposed. On the basis of RetinaNet, the algorithm uses ShuffleNetV2 as the feature extraction network. A dual-branch attention module channel attention module is proposed, which adopts the dual-branch structure and adaptive fusion and enhances the ability to extract the key features of the target in infrared images. To optimize the feature fusion, the algorithm integrates cross-scale connection and fast normalized fusion in some feature layers to enhance the multi-scale feature expression. The calibration factor is set to enhance the task interaction of classification and regression, and the accuracy of target classification and locating is increased. A series of experiments are conducted on a self-built infrared vehicle dataset to verify the effectiveness of the proposed algorithm. The detection accuracy of this algorithm for the self-built vehicle dataset is 92.9%, the number of parameters is 11.74×10⁶, and the number of floating-point operations is 24.35×10⁹. The algorithm exhibits better detection performance on the public dataset FLIR ADAS. Experimental results indicate that the algorithm has advantages in detection accuracy and model complexity, giving it good application value in multi-scale vehicle detection tasks in infrared scenes.