Cloud detection is of great significance for the application of remote sensing images. However, as for the existing cloud detection methods, there is limited research on the polarization information of remote sensing images, and their performance and generalization ability are also limited. To effectively utilize the polarization information of remote sensing images, a multimodal fusion remote sensing image cloud detection method based on depth learning is proposed and its preliminary experimental evaluation is conducted. In the method, the network is a three-parameter input stream architecture with an encoder-decoder structure, and the channel-spatial attention module is used to perfom multimodal fusion of reflectance and polarization features in remote sensing images. In the upsampling stage of the decoder, the iterative attention feature fusion method is used to fuse the high- and low-level feature maps. The evaluation experimental data set comes from Directional Polarization Camera (DPC) cloud products and cloud mask products. The evaluation results show that the proposed network model achieves good cloud detection performance, with a recognition accuracy of 93.91%.