A remote sensing image fusion method is proposed based on multi-stream fusion generative adversarial network with radiation index. This method extracts feature maps from the Multi-Spectral(MS) image and Panchromatic(PAN) image according to specific rules, and uses the generator sub-network to refine the input multi-source image, then fuses images through the main network of the generator after superimposing the features. According to the characteristics of multi-spectral and panchromatic images in the remote sensing field, the Modulation Transfer Function(MTF) is introduced into the discriminator to simultaneously discriminate the spectral information and spatial structure information of the fused images. In order to evaluate the effectiveness of the proposed method, its visual analysis is conducted and it is compared with other algorithms for objective evaluation. Experimental results show that this method is superior to other algorithms in visual effects and objective evaluation.