In the binary description algorithm (Oriented Fast and Rotated Brief, ORB), scale and rotation cause a great error in the registration, and the registration rate is low. Meanwhile, the RANdom Sample Consensus (RANSAC) algorithm has an instability issue. Therefore, in this study, a fast feature matching algorithm was presented based on ORB with RANSAC. First, the feature point extraction method was optimized to eliminate the influence of feature edges. After constructing a simplified pyramid scale-space model, the scale-space structure of the layered image was improved by reducing the number of generated image layers. Subsequently, the gradient direction was used to improve the main direction extraction mode of the traditional ORB algorithm, and the accuracy of the main direction of the feature angular point was improved. Finally, the RANSAC algorithm was improved by applying block random sampling, which improved the stability and accuracy of image registration. Experimental results reveal that the improved ORB and RANSAC fusion algorithm performance greatly improved in terms of scale and rotation registration, and higher registration precision is exhibited in comparison with traditional ORB. The scale registration accuracy is improved by 55.41%, and the rotational registration accuracy is improved by 26.66%. These results indicate that the proposed algorithm basically meets the accuracy and real-time requirements for fast and accurate registration of complex images.