Because the infrared image lacks certain texture information, most target detection networks cannot achieve great detection results for infrared images. This paper proposes a cross-domain fusion network structure that combines multiple modal for infrared target detection. Using image conversion network without pairing, modal conversion of existing infrared dataset to generate a pseudo-visible light dataset. Then, this paper proposes a dual-channel multi-scale feature fusion structure in the infrared domain and the pseudo-visible light domain, uses feature pyramid network to obtain the feature map of each mode, and performs dual-modal feature fusion for multi-scale features. Finally, in order to make up for the lack of texture in the fusion process, this paper proposes a soft weight distribution module. By splicing the parameterized source domain, target domain and fusion domain features, the network weight is assigned and optimized through learning, thereby improving the accuracy of feature extraction and target detection. The experimental results show that the method in this paper has better infrared target detection performance compared with the conventional method.