Heavy metals are scarce in soil, and it is difficult to identify their obvious characteristics in the soil spectrum. The previous soil heavy metal estimation methods have mostly applied statistical methods to find the characteristic bands, which cannot accurately explain the retrieval mechanism. It is therefore difficult to establish a universal model for soil heavy metal estimation. In order to investigate the influence of soil heavy metals in visible and near-infrared spectroscopy and analyze the retrieval mechanism of soil heavy metals, it is necessary to study the absorption characteristics of iron/manganese oxides, organic matter, clay minerals, etc. In this study, 80 soil samples were collected from the experimental field at Xuzhou, China. The spectra of the soil samples were measured with an Analytical Spectral Devices (ASD) field spectrometer. The soil heavy metal contents (Cr, Cd, Cu, Pb, and Zn) were determined by inductively coupled plasma-mass spectrometry. The soil spectra were processed by continuum removal. The absorption peaks related to heavy metals were around 480, 1 780, and 2 200 nm, which can be mainly attributed to iron/manganese oxides, organic matter, and clay minerals in the soil. The four spectral absorption characteristic parameters of Depth480, Depth1 780, Depth2 200, and Area2 200 were extracted at the positions of the absorption peaks. The variation trends of the parameters, along with the contents of the five heavy metals, were then analyzed. It was found that the four parameters were strongly correlated with the contents of the five heavy metals. Using a single variable to estimate the heavy metals, it was found that Depth480 had a higher estimation accuracy for Cr and Pb, and Area2 200 and Depth1 780 had a higher estimation accuracy for Cd, Cu, and Zn. The four spectral absorption characteristic parameters were implemented as independent variables, and the regression coefficients were obtained by ordinary least squares, ridge regression, and support vector regression. The heavy metal estimation model using the four spectral absorption characteristic parameters was stronger and more stable than those using only a single parameter. The best R2p (determination coefficient of prediction) values of the estimation models (Cr, Cd, Cu, Pb, and Zn) were 0.71, 0.84, 0.92, 0.80, and 0.89 respectively. The results suggest that Cr and Pb are easily adsorbed by iron/manganese oxides, while Cd, Cu, and Zn are more easily adsorbed by organic matter and clay minerals in this study area. The results of this study will provide a reference for researchers exploring the relationship between soil spectral characteristics and heavy metals.