Traditional infrared small target detection methods based on the human visual system can easily cause false alarm detection in complex backgrounds. Therefore, to address this issue, an accurate and robust infrared small target detection method called Double-layer Local Contrast Measure (DLCM) was proposed. First, a double-layer-diagonal gray difference contrast was proposed to enhance the visual saliency of the target and alleviate the impact on background clutter and noise. After that, an adaptive threshold segmentation was used to obtain the real target. The experimental results show that, compared with the mainstream detection methods based on human vision system, the background suppression factor of the proposed method is increased by an average of 9.3 times and the signal to clutter ratio gain is increased by an average of 7.8 times, which have better detection performance in different complex scenarios.