An novel infrared dim and small target detection algorithm, called J-MSF, based on multi-channel and multi-scale feature fusion was proposed, which solved the problem that the classical infrared dim and small target detection algorithm based on deep learning cannot detect because the target information disappeared in the upper receptive field. Firstly, a new multi-channel Janet structure was proposed to design the J-MSF backbone extraction framework. Secondly, a descending threshold feature pyramid pooling structure (DSPP) was exploited, and a multi-scale fusion detection strategy was conducted. Finally, the Gauss loss optimization function was designed. The experimental results show that the recall rate and the AP value of the proposed algorithm are improved by 9.07%, 9.89% and 1.67%, 3.16%, respectively, compared with those of YOLOv3 and YOLOv4 algorithms in "a dataset for infrared detection and tracking of dim and small aircraft targets underground/air background". The proposed algorithm can be effectively applied to infrared dim and small target detection, shows good robustness and adaptability, and is better than the state of the art algorithms.