In order to overcome the problems of tracking and registration based on a target point cloud in augmented reality, a robust Z-score hybrid tree registration algorithm is proposed. The noise is identified by the vertical distance from the point in the local neighborhood to the fitting plane and the distribution at normal point of the plane. The robustness of the Z-score is enhanced by utilizing the median absolute deviation; the hybrid tree algorithm is used to improve the efficiency of the nearest-point search. We demonstrate formulation by applying the proposed method to the imaging principle of augmented reality. The proposed algorithm is verified by using the point cloud dataset from a research group in Stanford University and real data. Experimental results show that, for the point cloud dataset with noise, the algorithm can maintain a certain accuracy while effectively improving the registration efficiency, which takes time about 5%-10% of that of the comparison algorithm.