Remainders with a certain size in the infrared detector assembly easily cause the diffraction phenomenon in the optical path of infrared detector in service, which would change the local luminous flux distribution on the focal plane, resulting in 'black spot' and 'Poisson bright spot ' on the image. In order to reduce the occurrence of such abnormal images, according to Fresnel diffraction theory, the remainder particles diameter satisfying the diffraction spot formation for several typical detector Dewar assemblies was calculated, and the relationship among the remainders dimension, Dewar structure, wavelength and diffraction spot was also analyzed. The results show that diffraction is more likely to happen for longer wavelength and smaller distance between remainders and focal plane; for different detector Dewar assembly, the position inclined to diffraction is the region where the distance denoted by "L" between remainders and focal plane satisfies the equation "L<L_C". Besides, combined with the production practice, some corresponding measures to control the remainders were offered. These conclusions in the article provide a reference for the infrared detector production design and engineering application.