It is a challenging task to track pedestrian accurately in complicated environment such as illumination, background variation, occlusion, noise and fast motion. Aiming at these problems, the tracing algorithm based on HSV color features and reconstruction by contributions is proposed. The proposed algorithm extracts the mixed color features of target in HSV space to generate the target template set within the particle filter framework. According to the influence of different regions on the tracking results, the contribution of the region is distributed. And it is introduced into the adaptive regularization model, and the region with the minimum reconstruction error is determined as the target to be tracked. In order to be more robust, the templates are updated in real time during the tracking progress. The average center error of tracking results and tracking success rate of 100 sequences tested in OTB are 0.6624 pixel and 0.4153, respectively, and the proposed algorithm has better performance than others. Experimental results show that the proposed algorithm can realize the continuous tracking for pedestrian in complex video scenes and is beneficial to be realized in the practice system with better robustness.