In order to improve the tactical index of weapon system, it is necessary to realize the rapid cooling of infrared detectors. In this paper, the factors affecting the cooling time of the detectors are listed by analyzing the structure of the detectors. The comparative experiments of cooling time are made separately by reducing the heat capacity of dewar cooling stage, reducing the heat loss, improving the thermal conductivity of the adhesives and adjusting the distance between the nozzle and the cooling stage. By analyzing the experimental results, the following conclusions are drawn: for conical metal dewar, the biggest factor affecting the cooling time is the heat capacity of the cooling stage, and the proportion of the cooling time reduction is close to that of the heat capacity reduction. The second biggest factor is the efficiency of heat exchange between the cooler and the dewar. Improving other factors can also reduce the cooling time of the detectors, but the effect is not significant. The conclusion provides a more direct reference for improving the cooling time of infrared detectors.