Hyperspectral imaging technology has great potential in the identification of crop varieties because it contains both image information and spectral information for the object. But so far most studies only used the spectral information, the image information has not been effectively utilized. In this study, hyperspectral images of single seed of three types including strong gluten wheat, medium gluten wheat, and weak gluten wheat were collected by near infrared hyperspectra imager, 12 morphological characteristics such as length, width, rectangularity, circularity and eccentricity were extracted, the average spectra of endosperm and embryo were acquired by the mask which was created by image segmentation. Partial least squares discriminant analysis (PLADA) and least squares support vector machine (LSSVM) were used to construct the classification model with image information, results showed that the binary classification accuracy between strong gluten wheat and weak gluten wheat could achieve 98%, for strong gluten wheat and medium gluten wheat, it was only 74.22%, which indicated that hyperspectral images could reflect the differences of varieties, but the accuracy might be poor when recognizing the varieties just by image information. Soft independent modeling of class analogy (SIMCA), PLSDA and LSSVM were used to established the classification model with spectral information, the classification effect of endosperm is slightly better than the embryo, it demonstrated that the grain shape could influence the classification accuracy. Then, we fused the spectral and image information, SIMCA, PLSDA and LSSVM were used to established the identification model, the fusion model showed better performance than the individual image model and spectral model, the classification accuracy which used the PLSDA raise from 96.67% to 98.89%, it showed that digging the morphological and spectral characteristics of the hyperspectral image could effectively improve the classification effect.