The spectral characteristics of soil are the external manifestation of physical and chemical properties in soil. Estimating soil organic matter content (SOMC) by visible-near infrared (VIS-NIR) diffuse reflectance spectroscopy could provide an important scientific basis for the rational development and utilization of land resources. However, the soil is a mixture of many substances, and its hyperspectral data have overlapping absorption of certain components (such as salt particles and minerals), and there are collinear problems between the bands, which bring great challenges for spectral analysis and modeling. Through the iterative operation, the spectral index method not only fully consider the synergy between the bands, but also has the function of minimizing the influence of irrelevant wavelengths. In addition, the method extends the spectral features from one dimension to multidimensional, and can easily detect and distinguish subtle absorption peak. In this study, 120 soil samples were collected from the Ebinur Lake Basin in Xinjiang, and SOMC and spectra were measured indoors. Hyperspectral data were preprocessed using first derivative (FD) and continuum removal (CR). Based on the existing two-band index, the third band was added, and the three-band spectral index (TBI) of three SOMCs was constructed by using the optimal band algorithm. The rationality of TBI was discussed from the spectral mechanism. Finally, according to the modeling effect of support vector machine (SVM), the accuracy of SOMC estimation by different dimensional spectral parameters was further compared. The research results showed that: (1) Spectral pretreatment technology could weaken the noise information in the reflection spectrum to some extent and highlighted more potential spectral information; (2) Through comparative analysis, the correlation of SOMC increased with the increase of the spectral information dimension, that was, TBI>two-band index>one-dimensional spectral parameters; (3) The newly developed TBI provided better estimation results than the two-band index and one-dimensional spectral parameters in the modeling and verification process of SOMC. The TBI-1 had the best estimation effect and the determination coefficient of the modeling set. (R2C) was 0.88, the decision coefficient (R2V) of the verification set was 0.85, and the relative analysis error (RPD) was 2.43. In summary, this study compared the response and modeling accuracy of different dimensional spectral parameters to SOMC. It was found that the three-band spectral index was an important parameter for evaluating SOMC and had good performance. In addition, the combination of TBI and SVM algorithm could weaken soil noise information, improved the prediction accuracy of SOMC, and had strong application potential in the estimation of other biochemical parameters of soil.