In the process of moving object tracking, in order to solve the problems that the object is easy to loss because of the occlusion, illumination fluctuation, scale variation and other factors, and the tracking window of the traditional Camshift algorithm is easy to diverge, a moving object tracking algorithm is proposed based on the fusion of optimized hidden Markov model (HMM) and the block feature matching. Firstly, the principal component analysis (PCA) combined with the feature position is used to reduce the dimension of the affine scale invariant feature transformation (ASIFT) features to generate PCA-ASIFT features which can retain the key information of the object. Then, the of the PCA-ASIFT features can be optimized by using the optimal feature positions of the particle filter. Finally, the object blocks are established by HSV histogram model and the different weights are assigned to different blocks and the integration block features matching, which can improve the Camshift algorithm to accomplish the moving object tracking. The experimental results show that the proposed algorithm can achieve better tracking effect of moving object in natural scenes, and it has bette robustness to occlusion, scale variation and so on.