In order to solve the problem of imaging drift in scanning electron microscope (SEM) that caused by electron beam drift, electromagnetic interference and other reasons, an image shift correction algorithm based on ORB (oriented FAST and rotated BRIEF) combing the PROSAC (progressive sample consensus) is proposed in this paper. Firstly, the ORB algorithm is used to detect the feature between the reference image and real-time image. Then the initial matching of the feature is implemented by using the Hamming distance and cross-matching. Moreover, the RANSAN (random sample consensus) optimization algorithm PROSAC is used to calculate the homography matrix between frames and the final exact homography matrix is re-iterated after eliminating exterior point. Finally, the SEM image drift is corrected in real time using the perspective transformation of the homography matrix. The experiments show that the proposed algorithm is high precision and satisfies the requirement of SEM real-time processing.