By self-adaptive feature selection, the traditional mean shift tracking algorithm was improved and its robustness was strengthened for object tracking in the complicated circumstance. Since one or two fixed features (such as the color) were usually selected for object tracking in traditional mean shift tracking algorithm, object tracking would be failure in the changeable circumstance. The remarkable and non-remarkable features were determined respectively by analyzing the distinguishing degree between candidates of object feature tracked and changeable background so that the most effective features were then selected to achieve object tracking in the complicated and changing circumstance. The reliability of the improved algorithm has been verified in serial experimental results of moving object tracking in different circumstance.