Hyperspectral images contain a wealth of feature information, and they have been widely used in urban feature classification in recent years. In the process of hyperspectral image classification, the extraction of spatial spectral features directly affects the classification accuracy. Traditional hyperspectral image feature extraction methods only use 4 or 8 neighborhood pixels for simple convolution processing, thus losing a lot of complex and effective information. Convolution neural network (CNN) can automatically extract spatial spectral features and retain the same spatial information of the image, and the network model is simplified. However, with the increase of network depth, the network classification will degenerate, and the network lacks complementarity of relevant information, which will affect the classification accuracy. In this paper, a hyperspectral residual network for feature classification is designed for the degradation problem. Firstly, define the residual network module of the low, medium and high three-layer structure with convolution kernels of 16, 32, and 64. Then, convolve the 3-layer output features with 64 1×1 convolution kernels to complete the dimension matching and feature map. Next, the global average pooling (GAP) of the feature map is generated to generate the feature vector for classification. Finally, the Large-Margin Softmax objective function is introduced to achieve hyperspectral image classification. The experiments were performed using hyperspectral images from the Indian Pines, University of Pavia, and Salinas regions. The primary bands of the hyperspectral image were extracted by PCA. With the sample set of batch training being 100, the initial learning rate being 0.1, the momentum being 0.9, the weight delay being 0.000 1, and the maximum number of training iterations being 2×104, when the sample sizes of the three data sets are set to be 25×25, 23×23 and 27×27, the network depth is 28, 32 and 28, the classification accuracy of the three data sets is the highest, and the average overall accuracy OA is 98.75%, the average accuracy AA is 98.1% and the average Kappa coefficient is 0.98. The experimental results show that the classification method based on residual network can get more affective features. It can improve the classification accuracy with the increase of the number of residual network layers and the fusion of complementary information of different network layer outputs; Large-Margin Softmax achieves intra-class compactness. Separation between classes further improves classification accuracy.