• Optics and Precision Engineering
  • Vol. 31, Issue 17, 2598 (2023)
Yingxu LIU1, Chunyu PU1, Diankun XU2, Yichuan YANG1, and Hong HUANG1,*
Author Affiliations
  • 1Key Laboratory of Optoelectronic Technology and Systems of the Education Ministry of China, Chongqing University, Chongqing400044, China
  • 2Measurement and Control Technology and Instrument major, College of Optoelectronic Engineering, Chongqing University, Chongqing400044, China
  • show less
    DOI: 10.37188/OPE.20233117.2598 Cite this Article
    Yingxu LIU, Chunyu PU, Diankun XU, Yichuan YANG, Hong HUANG. Lightweight deep global-local knowledge distillation network for hyperspectral image scene classification[J]. Optics and Precision Engineering, 2023, 31(17): 2598 Copy Citation Text show less

    Abstract

    To address the challenges of the complex spatial layouts of target scenes and inherent spatial-spectral information redundancy of HSIs, an end-to-end lightweight deep global–local knowledge distillation (LDGLKD) method is proposed herein. To explore the global sequence properties of spatial-spectral features, the vision transformer (ViT) is used as the teacher to guide the lightweight student model for HSI scene classification. In LDGLKD, pre-trained VGG16 is selected as the student model to extract local detail information. After collaborative training of ViT and VGG16 through knowledge distillation, the teacher model transmits the learned long-range contextual information to the small-scale student model. By combining the advantages of the two models through knowledge distillation, the optimal classification accuracy of LDGLKD on the Orbita HSI scene classification dataset (OHID-SC) and hyperspectral remote sensing dataset for scene classification (HSRS) reached 91.62% and 97.96%, respectively. The experimental results revealed that the proposed LDGLKD method presented good classification performance. In addition, the OHID-SC based on the remote sensing data obtained by the Orbita Zhuhai-1 satellite could reflect the detailed information of land cover and provide data support for HSI scene classification.
    R0=[Cemb,S1M,S2M,...,SNM]+Pemb(1)

    View in Article

    Rl'=MSA(LN(Rl-1))+Rl-1(l=1,...,L)(2)

    View in Article

    Rl=MLP(LN(Rl'))+Rl'   (l=1,...,L)(3)

    View in Article

    Ftea=LN(RL0)(4)

    View in Article

    [Q,K,V]=R[UQ,UK,UV](5)

    View in Article

    Watt=SoftmaxQKTdkV(6)

    View in Article

    MSA(R)=[W1att,W2att,...,Wtatt]W(7)

    View in Article

    GELU(x)=xφ(x)=x121+erfx2(8)

    View in Article

    erf(x)=(2/π)0xe-η2dη(9)

    View in Article

    μf=1Bi=1Baif(10)

    View in Article

    σf=1Bi=1B(aif-μf)2(11)

    View in Article

    a^f=af-μf(σf)2+ε(12)

    View in Article

    GAP(U)=1Y×Zy=1Yz=1Zuy,z,c(c=1,2,...,C)(13)

    View in Article

    Lfir=αLKD+(1-α)(LCtea+LCstu)(14)

    View in Article

    Lsec=2αLKD+(1-2α)LCstu(15)

    View in Article

    LKD=T2×KLdiv(QstuT,QteaT)(16)

    View in Article

    KLdiv(p||q)=p(x)logp(x)q(x)(17)

    View in Article

    QstuT(j)=exp(Fstuj/T)bexp(Fstub/T)(18)

    View in Article

    QteaT(j)=exp(Fteaj/T)bexp(Fteab/T)(19)

    View in Article

    Yingxu LIU, Chunyu PU, Diankun XU, Yichuan YANG, Hong HUANG. Lightweight deep global-local knowledge distillation network for hyperspectral image scene classification[J]. Optics and Precision Engineering, 2023, 31(17): 2598
    Download Citation