A Doppler asymmetric spatial heterodyne interferometer can be used to detect Doppler frequency shift of airglow spectrum lines of the O2 band, and thus the wind speed in the high-level atmosphere can be retrieved. The absolute phase frequency shift of interference fringes caused by wind speed is small, and the absolute phase drift caused by system error has significant influence on the wind retrieval precision. Different with the single arm interferometer, the dual-arm interferometer has the absolute phase of its interferograms affected not only by the field-widened prisms and gratings, but also by the thermal expansion of spacers used to produce optical path difference. Therefore, the absolute phase drifts at different temperatures are measured experimentally and simulated numerically, and the reason of absolute phase drift is analyzed. Based on this, a absolute phase drift correction method is proposed. The distance between two linear phase fitting curves under the conditions of zero wind speed and a given wind speed is calculated, the temperature dependent absolute phase drift is corrected, and the wind speed is retrieved accurately. Results show that the measured phase drift fits well with the simulated phase drift. After the absolute phase drift correction, the retrieved wind speed error is 3.5 m/s, which is greatly improved compared with the result before absolute phase drift correction.