At present, the research of hyperspectral inversion methods for heavy metals is mostly focused on single areas or not consider the influence of soil type on the inversion results. However, the differences of soil type and soil forming factors may have a certain degree in influence on the applicability of hyperspectral data-based inversion model of soil parameters. Hyperspectral remote sensing data proposed a method for inverting soil Zn metal content and considering soil types. It extracted the characteristic spectrum of soil spectrally active constituents with strong sorption and retention for the heavy metal from laboratory spectra of soil samples to enhance the inversion mechanism. For each soil type, the improved genetic algorithm (IGA) was employed on the characteristic spectrum to select the effective bands furtherly, and these bands were used to construct model by the partial least squares regression (PLSR). Finally, different modeling methods are evaluated using the coefficient of determination (R2), relative deviation (RPD) and root mean square error of prediction (RMSEP). The proposed method was validated by 38 yellow soil samples and 35 red soil samples collected in the Dong River lead-zinc mining area in Chenzhou City, Hunan province, and then the soil Zn content inversion model of each of the two soil types was constructed by the characteristic spectrum of organic matter and clay minerals extracted from the laboratory spectra and finally both were modeled by the IGA+PLSR. The results showed, when modeling with all soil samples regardless of soil types, comparing with inversion using the entire spectral range, the R2 and RPD were improved from 0.624 and 1.668 to 0.755 and 2.069 and the RMSEP decreased by 40.591 by using the characteristic spectrum associated with soil organic matter and clay minerals. When considering soil types and modeling respectively, comparing with the inversion without considering soil types, for yellow soil, the R2 was improved from 0.761 to 0.879, RPD was improved from 2.137 to 3.001, and the RMSEP decreased by 74.737; for red soil, the R2 was improved from 0.866 to 0.939, RPD was improved from 2.848 to 4.212, and the RMSEP decreased by 89.358. Inversion models for yellow and red soil samples met the criteria for an excellent model. Therefore, the proposed hyperspectral remote sensing inversion method of soil heavy metals content, which extracts the characteristic spectrum of soil spectrally active constituents and takes into account the soil type, is beneficial to improve the accuracy of heavy metal inversion and lays the foundation for monitoring of soil heavy metal pollution on a large scale by using hyperspectral remote sensing images.