The spatio-temporally modulated imaging Fourier transform spectrometers obtain two-dimensional spatial images modulated by interference pattern, and are sensitive to the change of attitude. According to the imaging principle of these instruments, image motion between two interference images should be one column of pixels when the instruments are in a vertical attitude, and abnormal image motion would arise because of geometric distortion and rotation of the interference images when an attitude incline exists. The expression of the abnormal image motion was deduced from the collinear equation. It was proved by numeric simulation that the abnormal image motion would accumulate in the interference image series and would make the objects′ interference pattern distorted. The interference images should be corrected to ensure the reality of the restored spectra in one of the following cases: vertical photography with platform of low pointing accuracy, oblique photography, low orientation accuracy of detector array, and aerial photography with a drift angle.