An image feature point matching algorithm based on the oriented fast and rotated brief (ORB) algorithm and hue, saturation and value (HSV) is proposed and the experimental research is carried out. Firstly, the image is preprocessed by the combination of bilateral filtering and mean filtering. Secondly, the ORB algorithm is used to extract feature points. Thirdly, the K-D Tree algorithm and Hamming distance are used for matching of feature points roughly, and then the HSV information of the image are used for the secondary screening of matched feature point pairs. The experimental results show that, in the image preprocessing stage, the weighted average of variance, vollath and information entropy is used as the evaluation index, and compared with the original image, histogram equalization and bilateral filtering results, the evaluation index value obtained by the combination of bilateral filtering and mean filtering is the best. In the stage of feature point matching and image mosaic, the average matching correct rate of feature points is improved by 12.60 percentage points after using HSV information, and the quality of image mosaic result is better, as its natural image quality evaluation (NIQE) index value is smaller.