To solve the mismatching between candidate model and reference model caused by state changing of the tracked target, a novel model updated method based on object prior knowledge was proposed in Mean Shift framework. The algorithm was implemented by using an auxiliary model constructed on color statistical knowledge of the tracked object under different states. This auxiliary model instructed the update of different color characters. Therefore, a reference model with tracked object prior information was formed. In such way, the time selection problem of model update was also solved by the proposed method. Experiment results under complex scenes show that the novel algorithm overcomes the mismatching of models caused by head rotation effectively and is able to realize continuous head tracking. In addition, the adaptive ability of Mean Shift algorithm is greatly improved.