In order to meet the space application requirements of the low-level infrared detector assembly in the high reliability field, environmental reliability tests, such as temperature cycling test, mechanical test and high temperature aging test had been carried out on the space-borne infrared detector assembly. Meanwhile, the relative spectral responsivity of the infrared detector assembly was tested based on double integral ball uniform illumination system before and after environmental reliability tests. By comparing differences of the relative spectral responsivity of the infrared detector assembly before and after environmental reliability tests, the environmental adaptability of the infrared detector assembly was analyzed, while the quality defects and other potential defects of the infrared detector assembly were revealed with the early eliminated failures. Then, the detectors with optimal performance were extracted from the participating infrared detector assembly, which were used in the space-borne polarization scanner for atmospheric polarization detection. The test results show that the tested infrared detector assembly are in good stability and reliability before and after environmental reliability tests, which can meet the aerospace load application requirements.