A hyperspectral image classification method is proposed based on a Gaussian linear process and multi-neighborhood optimization to overcome the poor classification accuracy of a classification algorithm based on spectral information. First, Gaussian filtering and linear discriminant dimension reduction are performed on the original sample data; then, the data are classified using a multivariate logistic regression model to obtain their initial prediction labels. Finally, the spatial position information of the local pixels is combined to determine the confidence of these prediction labels, which are corrected by the 3-layer tandem neighborhood optimization to obtain the final classification results. The proposed algorithm is compared with other algorithms on the Indian Pines, Pavia University, and Salinas hyperspectral remote sensing databases, demonstrating the enhanced performance in terms of classification accuracy and time efficiency of the proposed method.