When applying Spectral Histogram Representation (SHR) for texture classification, each image window is represented as a feature vector consisting of histograms of filtered images, and the histograms are concatenated to form the spectral representation for the image. Independent spectral representations and histograms for each image patch and each filter are combined to yield only very low-level local feature before the algorithm of filter selection. Finally, a filter selection algorithm is applied to select a small number of filters from all the independent filters. To improve classification reliability, aGaussian Radial Basis Function (RBF) is chosen on the Spectral Histogram Representation and the Support Vector Machines (SVMs) is used as classifying function. Comparison experiments between the proposed method and the other two methods:Gabor filtering and Independent Component Analysis (ICA) in texture classification and face recognition are performed.Experimental results demonstrate that higher categorization accuracy can be achieved with the proposed method, and theexcellence of the generalization performance of SVMs can be confirmed.