Making use of information in infrared images to build an effective observation model is the basis for realizing robust infrared target tracking. Besides the regular factors that have adverse influence on visual target tracking, infrared target tracking is faced with other difficulties as well, such as lack of edge and texture information, low signal-to-noise ratio and background clutter. An infrared target tracking algorithm based on histograms of sparse coding (HSC) and the distractor-aware model (DAM) is proposed, which exploits K singular value decomposition algorithm to obtain an overcomplete dictionary. With the dictionary, sparse code of every pixel is computed to compose HSC as a descriptor, and DAM is utilized to strengthen resistance against background clutter. The proposed algorithm does not only use structural information of tracked object but also eliminates the influence of background clutter. Compared with other tracking algorithms, the proposed algorithm achieves 3.8% and 4.4% enhancement on VOT-TIR2015 dataset with respect to precision and success rate, respectively, possessing high research and practical value.