There are two major problems of sophisticated vegetation classification (SVC) using hyperspectral image. Classification results using only spectral information can hardly meet the application requirements with the needed vegetation type becoming more sophisticated. And applications of classification image are also limited due to salt and pepper noise. Therefore the SVC strategy based on construction and optimization of vegetation feature band set (FBS) is proposed. Besides spectral and texture features of original image, 30 spectral indices which are sensitive to biological parameters of vegetation are added into FBS in order to improve the separability between different kinds of vegetation. And to achieve the same goal a spectral-dimension optimization algorithm of FBS based on class-pair separability (CPS) is also proposed. A spatial-dimension optimization algorithm of FBS based on neighborhood pixels’ spectral angle distance (NPSAD) is proposed so that detailed information can be kept during the image smoothing process. The results of SVC experiments based on airborne hyperspectral image show that the proposed method can significantly improve the accuracy of SVC so that some widespread application prospects like identification of crop species, monitoring of invasive species and precision agriculture are expectable.