The moving object detection is a prerequisite and difficult point in the video tracking system to realize tracking.In order to detect moving object effectively, an object tracking algorithm is proposed based on combining dynamics template matching and Kalman filter. First, make the former two frames inter-difference to get the area of the moving object and extract the feature points. Then, find the best match with the object model candidate object location by Kalman filter in the search area and match it with the object template of the current frame. Finally, the loss rate of feature points will serve as the limited threshold, and we update template according to dynamic template update strategy. Several experiments of the object tracking show that the approach is accurate and fast, and it has a better robust performance during the posture changing, the size changing and the shelter instance.