Spatial texture information extraction of hyperspectral image by filter often falls into local texture extraction. According to the problem, an algorithm of hyperspectral image classification based on adaptive manifold filtering (AMF-SVM) is proposed. This method uses adaptive optimization. The first manifold is calculated. Then, hyperspectral image with manifold is recursively splatted, blurred, and sliced according to the height of the manifold tree. Combined with the handling results, hyperspectral image is applicated to the linear filtering, the results are classified by support vector machine (SVM), and then the optimal classification is obtained. Experimental results show that the AMF-SVM algorithm is better than original SVM classification methods using the spectrum information, dimensionality reduction, and the spatial-spectral information, and the methods of edge-preserving filtering and recursive filtering. Performance of the classification for hyperspectral image with AMF-SVM is greatly improved, and effectiveness of this method is fully verified.