Aiming at the curse of dimensionality problem in measuring spectral similarity caused by the high dimensionality, high redundancy, non-linearity and small samples of the near-infrared spectrum, a local preserving projection algorithm based on kernel mapping and rank-order distance (KRLPP) is proposed in this paper. First, the spectral data is mapped to a higher-dimensional space through a kernel transformation, which effectively ensures the manifold structure’s nonlinear characteristics. Then, the dimensionality of the data is reduced by the locality preserving projections (LPP) algorithm, the rank-order distance is introduced instead of the traditional Euclidean distance or geodesic distance, and a more accurate local neighborhood relationship can be obtained by sharing the information of neighboring points. Finally, the measurement of the spectrum is realized by calculating the distance in low-dimensional space. This method solves the problem of distance failure in high-dimensional space and improves the accuracy of similarity measurement results. In order to verify the effectiveness of the KRLPP algorithm, firstly, the best parameters including the number k of the nearest neighbors and the dimensionality d of the reduced space were determined according to the residuals variation of the dataset before and after dimension reduction. Secondly, it compared with PCA, LPP, and INLPP algorithms from the perspectives of the projection effect of the spectra dimension reduction and the model classification ability. The results show that the KRLPP algorithm has a better ability to distinguish tobacco positions, and the effects of dimension reduction and correct identification of different tobacco positions are significantly better than PCA, LPP and INLPP methods. Finally, five representative tobacco were selected as target tobacco from a certain brand of cigarette formula. At the same time, PCA, LPP and KRLPP methods were used to find similar tobacco for each target tobacco from 300 tobacco samples used for formula maintenance, and the tobacco and cigarette formulas before and after replacement were evaluated from the aspects of chemical composition and sensory. Among them, the parameter selection of LPP and KRLPP for dimensionality reduction is consistent, and 6 principal components were selected for PCA. The results showed that, compared with PCA and LPP methods, the chemical components of total sugar, reducing sugar, total nicotine, total nitrogen and sensory indexes such as aroma, smoke and taste of the replacement tobacco and the replacement formula selected by the KRLPP algorithm had the least difference, and the accuracy of similarity measurement was the highest. This method can be applied to search for alternative raw materials for formula products and assist enterprises in maintaining product quality.