A dual channel residual convolution neural network with independent weight is established. The features of target in visible and infrared images is extracted.The multi-scale composite frequency band feature maps are generated. Based on the Euclidean distance between image points， the saliency of each image point in the dual band feature map is calculated. The adaptive fusion is carried out according to the characteristic contribution value of the target in different imaging frequency bands. Through the thermal radiation pooling kernel and visual attention mechanism， the logical mask of the target region of interest under dual frequency band is generated and superimposed on the fusion image to highlight the target features and suppress the non target area. Based on end-to-end identification network and using the cross loss calculation strategy. The target recognition of multi-scale dual band fusion feature map with attention mask is carried out.The results show that the designed recognition network can effectively integrate the physical characteristics of infrared heat source and the line features of visible image. The depth of information fusion is improved. The thermal radiation and texture features of the target is retained. The interference of background information is reduced. It has good recognition accuracy and robustness for multi-size heat source targets in all-weather and complex environment.