Covariance matrix is an excellent objet descriptor which can fuse multiple features and provide a global optimal resolution.Unfortunaely,it is hard for traditional covariance matching to track target when severe occlusion occurres.Further more,more similar background′s disturbances may be introduced by global search.In order to improve the performance of covariance tracking,a covariance tracking algorithm based on forgetting factor and Kalman filter is proposed.Multiple features can be skillfully fused using covariance matrix.In order to reduce the disturber from similar targets,weights on the distance function among covariance mattrixes is imposed using forgetting factor based on fuzzy membership.The Kalman filter is used to predict the trajectory of target and judge whether severe occlusions occurre which allow us to capture again when occlusions disappear.Experimental results show that the method can successfully cope with camera moving,clutter,occlusions,and target variations such as scale and rotation for tracking rigid and non-rigid targets.