To quickly detect the total organic carbon (TOC) content of oil shale, the TOC content and near-infrared spectrum data of 230 rock samples were measured in a certain block of the Songliao basin. The Monte Carlo method eliminates 14 abnormal samples, and the remaining 216 samples are pretreated by the method of detrended and baseline correction. The feature wavelength is selected by successive projections algorithm (SPA), uninformative variable elimination(UVE) algorithm and competitive adaptive reweighted sampling(CARS) method. The SPXY method divides the sample set into calibration set(144 samples) and validation set(72 samples) according to the ratio of 2:1. Then linear partial least squares (PLS) model, nonlinear support vector machine (SVM) model and random forest (RF) model are adopted to predict the TOC content of oil shale. The determination coefficient (R²) and root mean square error (RMSE) was used as the evaluation indexes of the model to explore the influence of different characteristic wavelength selection methods on TOC modeling of oil shale and to compare the accuracy of different modeling methods on TOC content prediction of oil shale. The results show that the feature wavelength extraction can optimize the model. SPA, UVE and CARS extract 16, 253 and 65 wavelength points respectively. After the feature wavelength extraction, the model determination coefficient is improved, and the root means square error is decreased. This shows that the feature wavelength extraction plays an important role in simplifying the model and improving model efficiency. In addition, The performance of the nonlinear RF and SVM model is better than that of the linear PLS model. The reason is that the carbon in oil shale exists in all kinds of hydrocarbons, and the absorption peaks of different hydrocarbon groups interact with each other, which makes the complex nonlinear relationship between the TOC content of oil shale and the near-infrared spectroscopy data. Therefore, the nonlinear SVM and RF model can show better performance. Compared with other models, the coefficient of determination ($R_{v}^{2}$) and root mean square error (RMSEV) of the CARS-SVM model invalidation set show better results, reaching 0.906 6 and 0.222 0 respectively. This model can be used to rapidly detect TOC content in oil shale. The results of this study show that the application of near-infrared spectroscopy in the rapid detection of TOC content in oil shale is feasible, and the CARS-SVM model can show good prediction performance, which provides a new method and idea for the rapid detection of TOC content in oil shale in China.