Soil organic carbon content was a major determinant of soil fertility and soil quality and was closely related to soil productivity. The estimation of soil organic carbon content using hyperspectral models has become an important method of understanding soil fertility. Using hyperspectral analysis combined with machine algorithms to achieve rapid and highly accurate estimation of soil organic carbon contents was essential for the sustainable use of soil fertility. Using the measured soil organic carbon content and its hyperspectral reflectance data as the research object, we applied the Savitzky Golay method to smooth and demise the spectral bands, used successive projection algorithm (SPA) and genetic algorithm (GA) to screen the original spectra and its five different mathematical transformed spectra respectively for the characteristic bands, and constructed the random forest (RF) method based on the soil organic carbon content. The hyperspectral estimation model of soil organic carbon content was constructed using the random forest (RF) method. The SPA algorithm was combined with the GA algorithm to find the optimal feature parameters to improve the recognition rate and confidence in the SOC feature bands. The results showed that in the original spectrum, the hyperspectral response bands based on the GA algorithm to screen SOC content were mainly concentrated on 350~410, 827~928, 997~1 064, 1 201~1 234, 1 541~1 574, 1 667~1 710, 2 153~2 186, 2 357~2 707 nm. When the RMSE was 6.09, 11 characteristic variables were screened by the SPA algorithm. The dimension of the original spectrum, standard normal variables (SNV), multiple scattering corrections (MSC), first-order differential (FD), logarithmic reciprocal (RL) and continuum removal (CR) were reduced to 407, 697, 668, 667, 493 and 784 dimensions respectively, accounting for 18.93%~36.47% of the full spectral band when filtering the characteristic bands based of the GA algorithm. After screening based on the GA-SPA algorithm, the dimensions of the six spectral variables ranged from 8 to 17 dimensions, and the RMSE ranged from 4.53 to 6.30. In the first-order differential spectral form, the RF model constructed based on 12 feature variables selected by the GA-SPA algorithm predicted the best results from a modeling set R2c of 0.78 and RMSEc of 5.48, a validation set R2p of 0.82, RMSEp of 4.50, and RPD of 2.18. It was shown that the first-order spectral differentiation could enhance the spectral information about soil, the GA algorithm combined with the SPA algorithm to find the spectral feature variables simplifies the complexity. It improves the accuracy of the estimation model, and the hyper spectral model based on the genetic algorithm-continuous projection algorithm has a high estimation capability.