Traditional hyperspectral image classification algorithms focus on spectral information application, however, with the increase of spatial resolution of hyperspectral remote sensing images, hyperspectral imaging presents clustering properties on spatial domain for the same category. It is critical for hyperspectral image classification algorithms to use spatial information in order to improve the classification accuracy. However, the marginal differences of different categories display more obviously. If it is introduced directly into the spatial-spectral sparse representation for image classification without the selection of neighborhood pixels, the classification error and the computation time will increase. This paper presents a spatial-spectral joint sparse representation classification algorithm based on neighborhood segmentation. The algorithm calculates the similarity with spectral angel in order to choose proper neighborhood pixel into spatial-spectral joint sparse representation model. With simultaneous subspace pursuit and simultaneous orthogonal matching pursuit to solve the model, the classification is determined by computing the minimum reconstruction error between testing samples and training pixels. Two typical hyperspectral images from AVIRIS and ROSIS are chosen for simulation experiment and results display that the classification accuracy of two images both improves as neighborhood segmentation threshold increasing. It concludes that neighborhood segmentation is necessary for joint sparse representation classification.