• Advanced Photonics
  • Vol. 6, Issue 1, 014002 (2024)
Xuemei Hu1、2, Weizhu Xu1、2, Qingbin Fan1、2, Tao Yue1、2, Feng Yan1、2, Yanqing Lu1、3, and Ting Xu1、3、*
Author Affiliations
  • 1Nanjing University, Collaborative Innovation Center of Advanced Microstructures, National Laboratory of Solid-State Microstructures, Nanjing, China
  • 2Nanjing University, School of Electronic Sciences and Engineering, Nanjing, China
  • 3Nanjing University, College of Engineering and Applied Sciences, Jiangsu Key Laboratory of Artificial Functional Materials, Nanjing, China
  • show less
    DOI: 10.1117/1.AP.6.1.014002 Cite this Article Set citation alerts
    Xuemei Hu, Weizhu Xu, Qingbin Fan, Tao Yue, Feng Yan, Yanqing Lu, Ting Xu. Metasurface-based computational imaging: a review[J]. Advanced Photonics, 2024, 6(1): 014002 Copy Citation Text show less
    Brief overview of the structure of the review.
    Fig. 1. Brief overview of the structure of the review.
    Achromatic computational imaging based on metasurface modulation techniques. EDoF-based achromatic imaging: (a) cubic phase mask, reproduced with permission from Ref. 30 (CC-BY), (b) symmetric EDoF-based canonical phase mask, reproduced with permission from Ref. 31 © 2020 Chinese Laser Press. Optimization-based inverse design methods: (c) multizone dispersion-engineered metalens-based achromatic RGB focusing, reproduced with permission from Ref. 32 (CC-BY), (d) achromatic RGB imaging with efficient 3D inverse design methods, reproduced with permission from Ref. 33 (CC-BY), (e) achromatic visible imaging with EDoF-based inverse design, reproduced with permission from Ref. 31 © 2021 American Chemical Society, and (f) achromatic inverse design based upon artificial neural network, reproduced with permission from Ref. 35 © 2021 Wiley-VCH.
    Fig. 2. Achromatic computational imaging based on metasurface modulation techniques. EDoF-based achromatic imaging: (a) cubic phase mask, reproduced with permission from Ref. 30 (CC-BY), (b) symmetric EDoF-based canonical phase mask, reproduced with permission from Ref. 31 © 2020 Chinese Laser Press. Optimization-based inverse design methods: (c) multizone dispersion-engineered metalens-based achromatic RGB focusing, reproduced with permission from Ref. 32 (CC-BY), (d) achromatic RGB imaging with efficient 3D inverse design methods, reproduced with permission from Ref. 33 (CC-BY), (e) achromatic visible imaging with EDoF-based inverse design, reproduced with permission from Ref. 31 © 2021 American Chemical Society, and (f) achromatic inverse design based upon artificial neural network, reproduced with permission from Ref. 35 © 2021 Wiley-VCH.
    Spectral modulation-based hyperspectral imaging. Hyperspectral imaging with random encoding with (a) regular-shaped metasurface design, reproduced with permission from Ref. 2 © 2022 Optical Society of America, (b) freeform-shaped metasurface, reproduced with permission from Ref. 44 © 2022 Wiley-VCH, (c) metasurface-based gratings, reproduced with permission from Ref. 45 © 2022 Optica Publishing Group, (d) multi-aperture spectral filter array-based hyperspectral imaging, reproduced with permission from Ref. 46 (CC-BY), (e) end-to-end learned optimal metasurface design, reproduced with permission from Ref. 3 © 2022 IEEE.
    Fig. 3. Spectral modulation-based hyperspectral imaging. Hyperspectral imaging with random encoding with (a) regular-shaped metasurface design, reproduced with permission from Ref. 2 © 2022 Optical Society of America, (b) freeform-shaped metasurface, reproduced with permission from Ref. 44 © 2022 Wiley-VCH, (c) metasurface-based gratings, reproduced with permission from Ref. 45 © 2022 Optica Publishing Group, (d) multi-aperture spectral filter array-based hyperspectral imaging, reproduced with permission from Ref. 46 (CC-BY), (e) end-to-end learned optimal metasurface design, reproduced with permission from Ref. 3 © 2022 IEEE.
    Polarization modulation-based computational metasurface imager. (a) Polarization multiplexing-based single-pixel imaging, reproduced with permission from Ref. 55 (CC-BY), (b) extreme-DoF imaging with polarization multiplexing, reproduced with permission from Ref. 59 (CC-BY), (c), (d) wide FoV microscopic imaging methods with polarization multiplexing, reproduced with permission from Ref. 58 (CC-BY) and Ref. 57 (CC-BY), (e) polarization multiplexing for underwater descattering, reproduced with permission from Ref. 56 © 2021 Wiley-VCH, (f) polarization multiplexing-based 4D imaging, reproduced with permission from Ref. 60 (CC-BY), (g) full-Stokes imaging, reproduced with permission from Ref. 4 © 2019 AAAS, (h) efficient polarization imaging with polarization splitting, reproduced with permission from Ref. 61 © 2020 Optical Society of America, (i) compressive polarization imaging with random weak dichroism metasurface, reproduced with permission from Ref. 5 (CC-BY).
    Fig. 4. Polarization modulation-based computational metasurface imager. (a) Polarization multiplexing-based single-pixel imaging, reproduced with permission from Ref. 55 (CC-BY), (b) extreme-DoF imaging with polarization multiplexing, reproduced with permission from Ref. 59 (CC-BY), (c), (d) wide FoV microscopic imaging methods with polarization multiplexing, reproduced with permission from Ref. 58 (CC-BY) and Ref. 57 (CC-BY), (e) polarization multiplexing for underwater descattering, reproduced with permission from Ref. 56 © 2021 Wiley-VCH, (f) polarization multiplexing-based 4D imaging, reproduced with permission from Ref. 60 (CC-BY), (g) full-Stokes imaging, reproduced with permission from Ref. 4 © 2019 AAAS, (h) efficient polarization imaging with polarization splitting, reproduced with permission from Ref. 61 © 2020 Optical Society of America, (i) compressive polarization imaging with random weak dichroism metasurface, reproduced with permission from Ref. 5 (CC-BY).
    Depth modulation and imaging techniques with metasurface imager. (a) DH-PSF engineering-based depth imaging, reproduced with permission from Ref. 65 (CC-BY), (b) EDoF and DH-based PSF engineering, with side-by-side metalens, reproduced with permission from Ref. 66 © 2020 ACS, (c) depth from dual-defocus multiplexing, inspired by jumping spider vision, reproduced with permission from Ref. 67 (CC-BY), and (d) triple metalens based 3D positioning, reproduced with permission from Ref. 68 © 2020 Optica Publishing Group.
    Fig. 5. Depth modulation and imaging techniques with metasurface imager. (a) DH-PSF engineering-based depth imaging, reproduced with permission from Ref. 65 (CC-BY), (b) EDoF and DH-based PSF engineering, with side-by-side metalens, reproduced with permission from Ref. 66 © 2020 ACS, (c) depth from dual-defocus multiplexing, inspired by jumping spider vision, reproduced with permission from Ref. 67 (CC-BY), and (d) triple metalens based 3D positioning, reproduced with permission from Ref. 68 © 2020 Optica Publishing Group.
    Angle dimension modulation for computational imaging. Wide-angle-imaging-based upon: (a) ommatidia-inspired pixel-wise angle-sensitive filtering, reproduced with permission from Ref. 70 (CC-BY), (b) angle-selective metalens array, reproduced with permission from Ref. 71 © 2022 Optica Publishing Group, (c) synthetic aperture with four small apertures, reproduced with permission from Ref. 72 © 2021 Chinese Laser Press. Wide-angle illumination for 3D depth imaging based upon: (d) pseudo-random coding, reproduced with permission from Ref. 73 (CC-BY), (e) uniform dense light patterns, reproduced with permission from Ref. 6 (CC-BY), (f) double-zone illumination, reproduced with permission from Ref. 74 (CC-BY), and (g) dual depth imaging mode with structured light-field imaging under common light conditions and structured imaging under low-light conditions, reproduced with permission from Ref. 75 © 2022 Wiley-VCH.
    Fig. 6. Angle dimension modulation for computational imaging. Wide-angle-imaging-based upon: (a) ommatidia-inspired pixel-wise angle-sensitive filtering, reproduced with permission from Ref. 70 (CC-BY), (b) angle-selective metalens array, reproduced with permission from Ref. 71 © 2022 Optica Publishing Group, (c) synthetic aperture with four small apertures, reproduced with permission from Ref. 72 © 2021 Chinese Laser Press. Wide-angle illumination for 3D depth imaging based upon: (d) pseudo-random coding, reproduced with permission from Ref. 73 (CC-BY), (e) uniform dense light patterns, reproduced with permission from Ref. 6 (CC-BY), (f) double-zone illumination, reproduced with permission from Ref. 74 (CC-BY), and (g) dual depth imaging mode with structured light-field imaging under common light conditions and structured imaging under low-light conditions, reproduced with permission from Ref. 75 © 2022 Wiley-VCH.
    Spectrum-angle/depth modulation-based computational imaging: spectral-angle joint modulation for (a) RGB and depth imaging with EDoF, reproduced with permission from Ref. 89 © 2021 ACS, (b) spectral light-field imaging, reproduced with permission from Ref. 91 (CC-BY), (c), (d) high-efficiency color imaging based upon color routing, reproduced with permission from Ref. 93 (CC-BY) and Ref. 92 © 2021 ACS, (e) single-image multichannel imaging, reproduced with permission from Ref. 94 © 2022 Optica Publishing Group, and (f) full-color wide-FoV imaging, reproduced with permission from Ref. 95 (CC-BY).
    Fig. 7. Spectrum-angle/depth modulation-based computational imaging: spectral-angle joint modulation for (a) RGB and depth imaging with EDoF, reproduced with permission from Ref. 89 © 2021 ACS, (b) spectral light-field imaging, reproduced with permission from Ref. 91 (CC-BY), (c), (d) high-efficiency color imaging based upon color routing, reproduced with permission from Ref. 93 (CC-BY) and Ref. 92 © 2021 ACS, (e) single-image multichannel imaging, reproduced with permission from Ref. 94 © 2022 Optica Publishing Group, and (f) full-color wide-FoV imaging, reproduced with permission from Ref. 95 (CC-BY).
    Computational imaging framework of metasurface-based imaging. (a) The computational imaging process, containing optimizable imaging component and reconstruction algorithms; (b) independent optimization framework; and (c) end-to-end optimization framework.
    Fig. 8. Computational imaging framework of metasurface-based imaging. (a) The computational imaging process, containing optimizable imaging component and reconstruction algorithms; (b) independent optimization framework; and (c) end-to-end optimization framework.
    Metasurface modulation-based computational sensing methods.
    Fig. 9. Metasurface modulation-based computational sensing methods.
    IlluminationSensingReconstruction
    Spectrum• Canonical EDoF phase-based achromatic design30,31• Wiener filtering36
    • Inversely-designed achromatic encoding34• TV-regularized convex optimization algorithm97,102,103
    • Random hyperspectral coding with regularly-shaped,2 free-form shaped,44 grating-based metasurface45• MLP-based spectral reconstruction3
    • Multiaperture hyperspectral filtering46• Dictionary learning-based sparse recovery48
    • PCA-based spectral encoding3
    • End-to-end learned metasurface design and neural network based hyperspectral retrieval and segmentation3,51
    Polarization• Trilobite-inspired chiral multiplexing59• Multiscale CNN59
    • Linear polarization multiplexing60• Matrix inversion4
    • Inversely-designed polarization splitter with tetrahedron polarization analyzer4• Gradient descent optimization algorithm23
    • Metalens array-based polarization splitting61• Deep mask-aware compressive full-Stokes reconstruction network5
    • Random polarization filtering5
    • Mosaic polarization filtering63
    Depth/Wide angle• High-density 1D or 2D dots (104) or parallel lines (100), covering 180 deg FoV6• Synthetic aperture sensing72• Truncated SVD technique76
    • M-array-based pseudo-random coding for illumination73• Ommatidia-inspired pixel-wise planar wide-angle selective sensing70• Wiener filtering36 and the Richardson–Lucy deconvolution algorithm77,78
    • Double-zone illumination, i.e., peripheral zone (150 deg FoV) and center zone (2 deg FoV)74• Angle-selective metalens linear array-based wide-angle sensing71• Mask-based stitching algorithm71
    • Depth extraction algorithm based on triangulation,87 stereo-matching,88 and time-of-flight74
    PSF engineering• DH-PSF65,66,69• Cepstrum analysis65
    • Jumping spider-inspired dual-defocus PSF67• Depth from defocus algorithm104
    • Three-subaperture based on three close-packed hexagonal metalenses68• Depth from parallex of correlation-aberration corrected subaperture images68
    Compound• Dispersion-based spectral encoding and metalens array-based light-field imaging91• Convex optimization of 4D hyperspectral light-field image based on spectral–spatial sparsity prior97
    • Inversely-design-based color routing or splitting92,93• U-Net-based RGB and depth retrieval neural network89
    • Tri-focus PSF over RGB channels89
    • End-to-end learned metasurface design and wide FoV imaging95 and snapshot multichannel (spectrum, polarization, and depth) imaging94
    Table 1. Overall summarization of existing computational metasurface imager.
    Xuemei Hu, Weizhu Xu, Qingbin Fan, Tao Yue, Feng Yan, Yanqing Lu, Ting Xu. Metasurface-based computational imaging: a review[J]. Advanced Photonics, 2024, 6(1): 014002
    Download Citation