In order to enhance the accuracy and the stability of attitude data, the basic attitude determination principle of star camera and gyroscope sensor was analyzed, and the attitude determination algorithm of star camera and gyroscope based on Unscented Kalman Filter (UKF) was derived with the error quaternion as the state variables. Because of the high-precision characteristics of star camera and gyroscope sensor, the UKF attitude determination experiments were carried out with the simulative star camera and gyroscope data of various precision, and were compared with the Extended Kalman Filter (EKF) attitude determination experiments. The results show that the UKF algorithm is effective and reliable, the attitude determination accuracy of star camera is effectively improved and the tri-axis accuracy is increased by about 10% to 20%.