When facing complex sea-sky background, island-shore background, bad weather, bright waves or decoys interference and other complex conditions, the detection rate, false alarm rate, detection distance or other performance indicators of the existing ship target detection system based on a single wide-wave infrared image will be affected. Considering the above problems, the detection method for ship target at sea based on multi-spectral infrared images was studied in this paper. Through the data acquisition system for multi-spectral infrared images, 107 groups of 5 medium-wave infrared images were collected actually. The spectrals from 1 to 5 were 3.7～4.8, 3.7～4.1, 4.4～4.8, 3.7～3.9 and 4.65～4.75 μm respectively. The sample data set was constructed by annotating the multi-spectral images manually, which was made up of 298 ship targets and 353 non-ship targets. Firstly, PCA transform was adopted to reduce the dimension of multi-spectral infrared images and selective search algorithm was adopted to generate the initial target candidate regions. In order to solve the problem that there are too many obvious non-ship target regions, the integral image was used to calculate the local contrast of the initial candidate regions and the ship target candidate regions were located according to the geometrical and grayscale features of infrared ship target. Secondly, each ship target candidate region was extended to incorporate the local context information. For the 5 spectral images corresponding to each ship target candidate region, dense SIFT feature of each spectral image was extracted. PCA was applied to SIFT feature, reducing its dimensionality from 128 to 64. Then the spatial and spectral position distribution information of each SIFT feature was added to the feature vector. Based on the Gaussian mixture model, the feature vectors of each candidate region were encoded to Fisher vector representation. Finally, linear SVM classifier was used to recognize ship targets. Experiment of the generation of ship target candidate regions showed that the proposed constraint method based on geometrical and grayscale features of infrared ship target can effectively overcome the shortcomings of selective search algorithm and quickly locate the ship target candidate regions from the initial target candidate regions. Experimental results on 25 groups of multi-spectral images showed that the generation of ship target candidate regions takes 0.353 s totally, while locating the ship target candidate regions takes only 0.005 s. Test of target recognition on 100 positive and negative samples showed that the recognition rate of the proposed target recognition algorithm reached 0.97, which is significantly higher than the target recognition rate based on single-wave infrared image. The proposed target recognition algorithm integrates the feature information of the multi-spectral target images and applies Fisher vector to extract the deep layer information in the gradient statistical features of the multi-spectral target images. Experimental results on 25 groups of multi-spectral images showed that the proposed ship target detection method can detect the ship targets at sea in different scenes such as sea-sky background, island background and bright waves interference. The locations of the ship targets are accurate and the ship recall rate reaches 0.95. The average detection time of each group of multi-spectral images is 1.33 s. The study results showed that with considering the radiation difference between the ship target and its local ocean background in the infrared image and the effective fusion of the radiation characteristics of ship target in multi-spectral infrared images, the divisibility of ship target can be enhanced, which results in the improvement of recognition rate and detection rate of ship target. This study provides new technical support for ship target detection based on multi-spectral infrared images.