Low-altitude unmanned aerial vehicle (UAV) measurement has become an important technical tool in the remote sensing field by the virtue of low-cost, high-efficiency, high-precision data acquisition mode and rapid acquisition images with high spatial resolution. Image matching technology is an important step in UAV image data processing, and the matching between images directly affects the accuracy and visual effect of the later 3D scenes. For the highland mountainous area, the topography is complex with large elevation changes, high vegetation cover and irregular distribution of features, making it difficult to match the images due to local noise in the UAV topographic survey processing. As the special terrain of the area limits the image acuqisition, large scene images need to be obtained by matching and stitching multiple images. At present, feature point-based image matching is an image alignment technique, which is applicable to the matching between low overlap images and can be applied to the matching between motion recovery images. To explore the fast and effective UAV image matching technique under special terrain and landscape conditions. This paper proposes an integration Scale Invariant Feature Transform (SIFT) algorithm, the Nearest Neighbor Distance Ratio (NNDR) algorithm and Random Sample Consensus (RANSAC) model constraints improved the UAV image matching method for complex terrain in highland mountains. The main technical process is as follows: Firstly, based on the SIFT algorithm for extreme value detection in scale space, a Gaussian pyramid function is constructed, and feature point localization is achieved by a Gaussian difference operation. It also performs statistical analysis on the neighborhood location, direction, and scale of the detected feature points to generate a description suitable for UAV image features. Secondly, the first constraint of feature pairs is extracted, and similarity is detected by integrating the “Mahalanobis distance” and NNDR models. On this basis, the RANSAC algorithm is used to introduce the root mean square error (RMSE) of the matched pairs for the second constraint, to achieve the rejection of the wrong matched pairs and ensure the accurate optimization of image matching. In addition, to confirm the effectiveness of the optimization algorithm proposed in this paper, one group of UAV image data of typical landscapes in the highland mountains were selected for matching tests. The results show that the improved algorithm proposed in this paper can extract a large number of point pairs and improve the correct detection rate of the same name points in UAV image matching for complex terrain in highland mountainous areas. Moreover, the correct rate of alignment reaches 85%, so it is more applicable to the optimization of UAV image matching processing technology for complex terrain in highland mountains.