It′s very difficult to calculate the calibration coefficients precisely using least mean squares (LMS) and extended Kalman filters (EKF) algorithm for wide field of view star sensor because of large optical aberration. And the error factors that affect measuring precision of star sensors are analyzed, which indicate that it is important to calibrate optical distortion before focal length calibration by comparing simulation data with and without distortion and noise. At the same time, four kinds of optical distortion methods are introduced. In order to improve calibration precision, a new method is proposed, in which main point is calibrated firstly, optical distortion parameter secondly and focal length finally. Simulation results demonstrate that the focal length calibration deviation reaches 3.1 μm and 2.2 μm using LMS and EKF algorithm after optical aberration correction. The statistical error of star angular distance is about 1/10~1/8 of the traditional on-orbit calibration method.