In flight, the installation matrix of star tracker changed with the orbital period affected by the space thermal environment. In order to calibrate the variation of the installation matrices between star trackers caused by the structural distortion of the satellite, a real time on-orbit installation matrix calibration method based on quaternion adaptive Kalman filter was proposed. The algorithm was based on the combination of fading memory filter and simplified Sage_Husa adaptive filter. The algorithm adjusted the weight of the current measurement by adjusting the fading factor adaptively, which could restrain the filter divergence caused by the inaccurate model parameters. The results of simulation experiments and on-board data verification show that the q-AKF algorithm not only could restrain the divergence problem caused by the inaccurate model parameters, but also could track the real values of the installation matrix stably, when the attitude maneuver rates ranged from 0 to 5 (°)/s. It is better in adaptability and robustness.