Shadows, mixed pixels and spectral variations are common in remote sensing images in mountainous karst areas due to complex terrain and broken surface. Dimidiate pixel model (DPM) based on multispectral remote sensing is difficult to accurately extract rocky karst desertification (KRD) information in areas with significant spectral variations and shadows. The mixed pixel decomposition technology of hyperspectral remote sensing can decompose complex mixed pixels into the mixed ratio corresponding to the pure landmark spectrum and each landmark spectrum, which provides the possibility for obtaining higher precision rocky desertification information in complex mountainous areas. However, due to the changes in many factors such as illumination, environment and atmosphere, the end members will vary to varying degrees, which will result in significant errors in the process of mixed pixel decomposition. Secondly, it is difficult to directly obtain the pure landmark spectrum from mountain images with complex terrain and strong surface heterogeneity and establish a spectrum library to deal with spectral variation. Therefore, the focus of current studies is how to deal with spectral variation and terrain effect in this case and obtain effective and accurate information extraction of rocky desertification. In order to solve the above problems, the generalized linear mixed model (GLMM), which simulates the reflectivity change of ground objects caused by illumination conditions and considers the spectral variation at each wavelength interval, was adopted to reduce the influence of spectral variation and terrain effect in the process of information extraction of rocky desertification in karst areas. First of all, the typical representative spectra of main ground objects (vegetation, bare rock and bare soil) in the karst area were extracted from GF-5 hyperspectral images. Then the spectral variation of each pixel under different illumination was simulated based on the extracted landmark spectrum, and the most suitable spectral combination was selected to decompose the pixels to get the best unmixing effect. In order to verify the reliability of the method, the visual interpretation results of high-resolution images were used as a reference to verify the prediction results of the method, and the fully constrained least squares linear spectral unmixing (FCLSU) DPM without considering end-member variation were selected for comparison. The results showed that it was necessary to consider shadows, mixed pixels and spectral variation in karst mountainous areas with highly complex terrain. The total accuracy of GLMM in rocky desertification information extraction reached 84.89%, significantly higher than that of the other two methods (59.68% and 67.34%). The accuracy of GLMM in the illumination area and shadow area was similar to that of GLMM in the illumination area and shadow area. However, the other two were quite different, and the shadow area was lower than the illumination area, which reflects that GLMM can effectively reduce the influence of terrain effect and improve the accuracy of information extraction of rocky karst desertification.