Infrared small target detection is widely used in aerial target detection and tracking systems owing to its long detection range and strong anti-jamming ability. Aiming at to overcome the shortcomings of the current infrared small target detection algorithm, such as a low precision rate and high false alarm rate when dealing with complex backgrounds, we propose an end-to-end infrared small target detection model (called MFSSD) based on multi-scale feature fusion. Considering the traits of the targets, we propose a feature fusion module using a refinement and fusion feature map method and improve the correlation of different channels through the SP module. The experimental results of three different sequences of infrared image detection show that the average detection accuracy of the MFSSD algorithm for infrared small target detection was as high as 87.8%. Compared with those of the traditional multi-scale target detection algorithm, both the precision rate and recall rate have been significantly improved.