Particle filter has already been extensively applied to video object tracking, however the traditional visual tracking approaches based on particle filter, which employ an experiential state transition equation and observation likelihood model predefined in advance, cannot satisfy the request of the real time complex situation video object tracking. In order to improve the robustness and stability as well as the computation efficiency of the video tracker based on particle filter, the adaptive state evolution equation and an online increment learning observation likelihood model are embedded into the particle filter, and the strategy for online self-adjusting the number of particle is adopted to enhance the computation efficiency. The experimental results show that the approach proposed in this paper not only track the moving object in the video accurately and effectively, but has nice robustness to the appearance variation caused by illumination and pose changes.