Recently, deep convolutional neural networks have made remarkable progress in visible light ship detection. However, most improve detection performance by improving large network structures, which require strong computer performance. In addition, the visible image is difficult to detect ships in the cloud, fog, sea clutter, night and other complex scenes. In order to solve these problems, this paper proposes a lightweight ship detection algorithm, which integrates the spectral information of Red, Green, Blue and NIR bands from coarse to fine. This paper uses the improved water detection algorithm to extract the ship candidate area from the existing methods that extract the water area by using the water detection algorithm according to the spectral characteristics. To obtain a more accurate candidate area, in this paper, the ships, the thick clouds, cloud, calm sea, five kinds of cluttered sea four bands of pixels in the scene have carried on the statistical analysis. Near-infrared is greater than the threshold as the auxiliary judgment, and in its center for candidate area the size of 32×32 slices, and to the maximum inhibition of slice, thus obtained the coarse detection results of the ship. Then constructs a lightweight LSGFNet network for fine identification of ship candidate regions. In the structural design of the network, the spectral features extracted by 1×1 convolution and the geometric features extracted by 3×3 are fused. In order to prevent the “information not flowing” during the fusion of spectral features and geometric features, the channel disruption mechanism in ShuffleNet is introduced in the LSGFNet network, and the model structure is reduced. Compared with the typical lightweight network, it has a better effect and a smaller model. Finally, 1 120 sets of data with 512×512 and 6014 sets of data with 32×32 were constructed for rough detection and fine network training using sentinel-2 multi-spectral 10-meter resolution data. Among them, the recall rate of rough extraction of candidate regions was 98.99%, and the fine identification network precision was 96.04%. The overall average precision is 92.98%. Experimental results show that the proposed algorithm has high detection efficiency in the complex background of suppressing clouds, sea clutter and other disturbances, the training time is short, and the computer performance demand is low.