Object tracking is a challenging research topic, which is widely used in infrared imaging search, infrared precision guidance, intelligent surveillance, motion recognition and other fields. In this paper, a robust template patches-based target tracking method with sparse representation was proposed. Firstly, the adaptive template patches selection mechanism was proposed using the discriminative information to capture the target. Then, the sparse representation was introduced to describe the patches to deal with the shortcoming of histogram′s sensitivity to light, which expanded the application of the algorithm. Thirdly, the target location was voted and fused by constructing a voting map. Finally, a dynamic updating scheme of patches was proposed to address appearance variations. The simulation experiments of test image sequences demonstrate the robustness of the proposed tracker, which is able to deal with many challenges, such as deformation, changes of illumination, partial and total occlusions, false target jamming and background interference.