According to the characteristics of infrared imaging and the electroluminescent principle of solar module, a detection system based on constrained independent component analysis (ICA) model and particle swarm optimization (PSO) method for solar module surface defects is proposed. Firstly, a constrained ICA filter is designed on the basis of the structure characteristics of infrared image for solar cells, and PSO method with multidirectional search characteristics is used to find the best demixing matrix for ICA. The constraint incorporated in the ICA model confines the source values of all training image patches of a defect-free image within a small interval of control limits and highlights the value of defect region. Then, ICA filter is used to convolve with the image. Finally, defect results are obtained through threshold segmentation. Experimental result shows that the proposed method can effectively detect the defects on solar module surface.