In this paper, we propose a cascade multi-scale information fusion generative adversarial network (CMIF-GAN) for infrared image simulation, which can estimate the infrared map from a visible image. Inspired by the connections and differences between visible and infrared features, CMIF-GAN adopts a cascaded structure composed of two levels of adversarial networks. With a large overall receptive field, the first-level adversarial network focuses on reconstructing structural information of the infrared image, and adds a semantic segmentation image task as auxiliary information. To enrich detailed texture information of the infrared image, the second-level adversarial network uses the grayscale inverted visible (GIV) images as auxiliary information and adopts a small overall receptive field network. Otherwise, the second-level adversarial network can integrate the multiple receptive information by a multi-scale fusion module (MFM) to improve algorithm accuracy. Experiments on public dataset demonstrate that CMIF-GAN can efficiently translate visible images to corresponding infrared images, and outperform previous methods in objective metrics and subjective vision.