The L1 tracker has good robustness towards partial occlusion, but the L1 tracker is sensitive to the outliers from the target templates and has slow computation speed. Aiming at these two problems, we propose a two-stage sparse representation model and design a relevant fast solution algorithm based on the block coordinate optimization theory. At the first stage, the algorithm uses the locality-constrained linear coding to solve the coefficients of the target templates. At the second stage, the algorithm uses the soft shrinkage operator to solve the coefficients of the trivial templates. Based on particle filtering method, the representation model and the algorithm are combined to achieve the robust fast visual tracking. The standard image sequences are used to verify the proposed method, and the results of the experiment show that the proposed tracking method outperforms the state-of-the-art trackers in terms of the robustness and the tracking speed.