As for the problems that question that color’s effectiveness had been drastically influenced by the variety of illumination and pose and interference of background in image tracking which will lead to tracking failure, a face tracking algorithm based on particle filter is proposed, which uses color histogram and contour distributions as its clues. There are three characteristics in this algorithm. Firstly, based on the basic framework of particle filter, a new method for describing the outline of face by histogram was proposed. It could reduce the influence when illumination, face rotation and partial face occlusions happened during tracking. Meanwhile, it could capture the lost target caused by large parts of occlusions quickly. At the same time, the feature point number in our method was adaptable which had improved the tracking efficiency. Secondly, for reducing the influence of the background, a method for suppressing the interference of similar background was proposed. Thirdly, the template was updated to improve the robustness of tracker during tracking. Experimental results show that our proposed algorithm could perform better than conventional tracker.