• Advanced Photonics
  • Vol. 6, Issue 1, 014002 (2024)
Xuemei Hu1、2, Weizhu Xu1、2, Qingbin Fan1、2, Tao Yue1、2, Feng Yan1、2, Yanqing Lu1、3, and Ting Xu1、3、*
Author Affiliations
  • 1Nanjing University, Collaborative Innovation Center of Advanced Microstructures, National Laboratory of Solid-State Microstructures, Nanjing, China
  • 2Nanjing University, School of Electronic Sciences and Engineering, Nanjing, China
  • 3Nanjing University, College of Engineering and Applied Sciences, Jiangsu Key Laboratory of Artificial Functional Materials, Nanjing, China
  • show less
    DOI: 10.1117/1.AP.6.1.014002 Cite this Article Set citation alerts
    Xuemei Hu, Weizhu Xu, Qingbin Fan, Tao Yue, Feng Yan, Yanqing Lu, Ting Xu. Metasurface-based computational imaging: a review[J]. Advanced Photonics, 2024, 6(1): 014002 Copy Citation Text show less
    References

    [1] K. Mohammadreza et al. Metalenses at visible wavelengths: diffraction-limited focusing and subwavelength resolution imaging. Science, 352, 1190-1194(2016).

    [2] X. Jian et al. Dynamic brain spectrum acquired by a realtime ultraspectral imaging chip with reconfigurable metasurfaces. Optica, 9, 461-468(2022).

    [3] M. Maksim et al. Real-time hyperspectral imaging in hardware via trained metasurface encoders, 12692-12702(2022).

    [4] A. Rubin Noah et al. Matrix Fourier optics enables a compact full-Stokes polarization camera. Science, 365, eaax1839(2019).

    [5] Q. Fan et al. Disordered metasurface enabled single-shot full-Stokes polarization imaging leveraging weak dichroism. Nat. Commun., 14, 7180(2023).

    [6] G. Kim et al. Metasurface-driven fullspace structured light for three-dimensional imaging. Nat. Commun., 13, 5920(2022).

    [7] M. G. L. Gustafsson. Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy. J. Microsc., 198, 82-87(2000).

    [8] M. J. Rust, M. Bates, X. Zhuang. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM). Nat. Methods, 3, 793-796(2006).

    [9] G. Zheng, R. Horstmeyer, C. Yang. Wide-field, high-resolution Fourier ptychographic microscopy. Nat. Photonics, 7, 739-745(2013).

    [10] J. K. Adams et al. In vivo lensless microscopy via a phase mask generating diffraction patterns with high-contrast contours. Nat. Biomed. Eng., 6, 617-628(2022).

    [11] B. Vivek et al. Lensless imaging: a computational renaissance. IEEE Signal Process Mag., 33, 23-35(2016).

    [12] M. O’Toole, D. B. Lindell, G. Wetzstein. Confocal non-line-of-sight imaging based on the light-cone transform. Nature, 555, 338-341(2018).

    [13] A. Velten et al. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nat. Commun., 3, 745(2012).

    [14] T. E. Bishop, S. Zanetti, P. Favaro. Light field superresolution, 1-9(2009).

    [15] P. Llull et al. Coded aperture compressive temporal imaging. Opt. Express, 21, 10526-10545(2013).

    [16] E. Reinhard et al. High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting(2010).

    [17] A. Levin et al. Image and depth from a conventional camera with a coded aperture. ACM Trans. Graph., 26, 70-es(2007).

    [18] G. R. Arce et al. Compressive coded aperture spectral imaging: an introduction. IEEE Signal Process. Mag., 31, 105-115(2013).

    [19] K. Akiyama et al. First M87 event horizon telescope results. IV. Imaging the central supermassive black hole. Astrophys. J. Lett., 875, L4(2019).

    [20] A. Velten et al. Femto-photography: capturing and visualizing the propagation of light. ACM Trans. Graphics, 32, 44(2023).

    [21] D. J. Brady et al. Multiscale gigapixel photography. Nature, 486, 386-389(2012).

    [22] J. N. Mait, G. W. Euliss, R. A. Athale. Computational imaging. Adv. Opt. Photonics, 10, 409-483(2018).

    [23] S. Ruder. An overview of gradient descent optimization algorithms(2016).

    [24] S. P. Boyd, L. Vandenberghe. Convex Optimization(2004).

    [25] G. Barbastathis, A. Ozcan, G. Situ. On the use of deep learning for computational imaging. Optica, 6, 921-943(2019).

    [26] E. R. Dowski, W. T. Cathey. Extended depth of field through wave-front coding. Appl. Opt., 34, 1859-1866(1995).

    [27] W. Chi, N. George. Electronic imaging using a logarithmic asphere. Opt. Lett., 26, 875-877(2001).

    [28] Z. Zhai et al. Extended depth of field through an axicon. J. Mod. Opt., 56, 1304-1308(2009).

    [29] N. Patwary et al. Experimental validation of a customized phase mask designed to enable efficient computational optical sectioning microscopy through wavefront encoding. Appl. Opt., 56, D14-D23(2017).

    [30] S. Colburn, A. Zhan, A. Majumdar. Metasurface optics for full-color computational imaging. Sci. Adv., 4, eaar2114(2018).

    [31] L. Huang et al. Design and analysis of extended depth of focus metalenses for achromatic computational imaging. Photonics Res., 8, 1613-1623(2020).

    [32] Z. Li et al. Meta-optics achieves RGB-achromatic focusing for virtual reality. Sci. Adv., 7, eabe4458(2021).

    [33] Z. Li et al. Inverse design enables large-scale high-performance meta-optics reshaping virtual reality. Nat. Commun., 13, 2409(2022).

    [34] E. Bayati et al. Inverse designed extended depth of focus meta-optics for broadband imaging in the visible. Nanophotonics, 11, 2531-2540(2021).

    [35] F. Wang et al. Visible achromatic metalens design based on artificial neural network. Adv. Opt. Mater., 10, 2101842(2022).

    [36] R. C. Gonzales, P. Wintz. Digital Image Processing(1987).

    [37] Bayesian methods in nonlinear digital image restoration. IEEE Trans. Comput., C-26, 219-229(1977).

    [38] P. Getreuer. Total variation deconvolution using split Bregman. Image Process. On Line, 2, 158-174(2012).

    [39] R. Pestourie et al. Inverse design of large-area metasurfaces. Opt. Express, 26, 33732-33747(2018).

    [40] K. Svanberg. A class of globally convergent optimization methods based on conservative convex separable approximations. SIAM J. Optim., 12, 555-573(2002).

    [41] J. Kennedy, R. Eberhart. Particle swarm optimization, 1942-1948(1995).

    [42] Y. Zhao et al. Hyperspectral imaging with random printed mask, 10149-10157(2019).

    [43] M. Yako et al. Video-rate hyperspectral camera based on a CMOS-compatible random array of Fabry–Perot filters. Nat. Photonics, 17, 218-223(2023).

    [44] J. Yang et al. Ultraspectral imaging based on metasurfaces with freeform shaped meta-atoms. Laser Photonics Rev., 16, 2100663(2022).

    [45] Z. Wu et al. Random color filters based on an all-dielectric metasurface for compact hyperspectral imaging. Opt. Lett., 47, 4548-4551(2022).

    [46] A. McClung et al. Snapshot spectral imaging with parallel metasystems. Sci. Adv., 6, eabc7646(2020).

    [47] M. F. Duarte, Y. C. Eldar. Structured compressed sensing: from theory to applications. IEEE Trans. Signal Process., 59, 4053-4085(2011).

    [48] S. Zhang et al. A spectral reconstruction algorithm of miniature spectrometer based on sparse optimization and dictionary learning. Sensors, 18, 644(2018).

    [49] M. E. Tipping, C. M. Bishop. Probabilistic principal component analysis. J. R. Stat. Soc. Ser. B Stat. Methodol., 61, 611-622(1999).

    [50] F. Getman et al. Broadband vectorial ultrathin optics with experimental efficiency up to 99% in the visible region via universal approximators. Light Sci. Appl., 10, 47(2021).

    [51] Q. Zhang et al. End-to-end joint optimization of metasurface and image processing for compact snapshot hyperspectral imaging. Opt. Commun., 530, 129154(2023).

    [52] D. S. Jeon et al. Compact snapshot hyperspectral imaging with diffracted rotation. ACM Trans. Graphics, 38, 117(2019).

    [53] L. Wang et al. Hyperspectral image reconstruction using a deep spatial-spectral prior, 8032-8041(2019).

    [54] C.-H. Lin et al. Metasurface-empowered snapshot hyperspectral imaging with convex/deep (CODE) small-data learning theory. Nat. Commun., 14, 6979(2023).

    [55] H.-C. Liu et al. Single-pixel computational ghost imaging with helicity-dependent metasurface hologram. Sci. Adv., 3, e1701477(2017).

    [56] F. Zhao et al. Metalens-assisted system for underwater imaging. Laser Photonics Rev., 15, 2100097(2021).

    [57] X. Ye et al. Chip-scale metalens microscope for wide-field and depth of-field imaging. Adv. Photonics, 4, 046006(2022).

    [58] B. Xu et al. Metalens-integrated compact imaging devices for wide-field microscopy. Adv. Photonics, 2, 066004(2020).

    [59] Q. Fan et al. Trilobite-inspired neural nanophotonic light-field camera with extreme depth-of-field. Nat. Commun., 13, 2130(2022).

    [60] Z. Shen et al. Monocular metasurface camera for passive single-shot 4D imaging. Nat. Commun., 14, 1035(2023).

    [61] M. Miyata, M. Nakajima, T. Hashimoto. Compound-eye metasurface optics enabling a high-sensitivity, ultra-thin polarization camera. Opt. Express, 28, 9996-10014(2020).

    [62] D. P. Bertsekas. Constrained Optimization and Lagrange Multiplier Methods(2014).

    [63] J. Zuo et al. Chip-integrated metasurface full-Stokes polarimetric imaging sensor. Light Sci. Appl., 12, 191(2023).

    [64] A. Greengard, Y. Y. Schechner, R. Piestun. Depth from diffracted rotation. Opt. Lett., 31, 181-183(2006).

    [65] C. Jin et al. Dielectric metasurfaces for distance measurements and three-dimensional imaging. Adv. Photonics, 1, 036001(2019).

    [66] S. Colburn, A. Majumdar. Metasurface generation of paired accelerating and rotating optical beams for passive ranging and scene reconstruction. ACS Photonics, 7, 1529-1536(2020).

    [67] Q. Guo et al. Compact single-shot metalens depth sensors inspired by eyes of jumping spiders. Proc. Natl. Acad. Sci. U. S. A., 116, 22959-22965(2019).

    [68] W. Liu et al. Aberration-corrected three-dimensional positioning with a single-shot metalens array. Optica, 7, 1706-1713(2020).

    [69] Z. Ma et al. Reconfigurable metalens with phase-change switching between beam acceleration and rotation for 3D depth imaging. Micromachines, 13, 607(2022).

    [70] L. C. Kogos et al. Plasmonic ommatidia for lensless compound-eye vision. Nat. Commun., 11, 1637(2020).

    [71] J. Chen et al. Planar wide-angle-imaging camera enabled by metalens array. Optica, 9, 431-437(2022).

    [72] F. Zhao et al. Synthetic aperture metalens. Photonics Res., 9, 2388-2397(2021).

    [73] X. Jing et al. Single-shot 3D imaging with point cloud projection based on metadevice. Nat. Commun., 13, 7842(2022).

    [74] R. J. Martins et al. Metasurface-enhanced light detection and ranging technology. Nat. Commun., 13, 5724(2022).

    [75] M. K. Chen et al. A meta-device for intelligent depth perception. Adv. Mater., 35, 2107465(2022).

    [76] P. C. Hansen, T. Sekii, H. Shibahashi. The modified truncated SVD method for regularization in general form. SIAM J. Sci. Stat. Comput., 13, 1142-1150(1992).

    [77] W. H. Richardson. Bayesian-based iterative method of image restoration. J. Opt. Soc. Am., 62, 55-59(1972).

    [78] L. B. Lucy. An iterative technique for the rectification of observed distributions. Astron. J., 79, 745(1974).

    [79] J. Hu et al. Polarization-insensitive, orthogonal linearly polarized and orthogonal circularly polarized synthetic aperture metalenses. Photonics, 10, 348(2023).

    [80] C.-Y. Yu et al. Optical performance of synthetic aperture metalens based on hybrid unit-cells. Mater. Sci. Semicond. Process., 170, 107982(2024).

    [81] J. N. Gollub et al. Large metasurface aperture for millimeter wave computational imaging at the human-scale. Sci. Rep., 7, 42650(2017).

    [82] J. Hunt et al. Metamaterial apertures for computational imaging. Science, 339, 310-313(2013).

    [83] M. Boyarsky et al. Synthetic aperture radar with dynamic metasurface antennas: a conceptual development. J. Opt. Soc. Am. A, 34, A22-A36(2017).

    [84] Y. Ni et al. Metasurface for structured light projection over 120 field of view. Nano Lett., 20, 6719-6724(2020).

    [85] J. F. MacWilliams, N. J. A. Sloane. Pseudo-random sequences and arrays. Proc. IEEE, 64, 1715-1729(1976).

    [86] R. W Gerchberg. A practical algorithm for the determination of plane from image and diffraction pictures. Optik, 35, 237-246(1972).

    [87] O. Faugeras. Three-Dimensional Computer Vision: A Geometric Viewpoint(1993).

    [88] A. Myronenko, X. Song. Point set registration: coherent point drift. IEEE Trans. Pattern Anal. Mach. Intell., 32, 2262-2275(2010).

    [89] S. Tan et al. 3D imaging using extreme dispersion in optical metasurfaces. ACS Photonics, 8, 1421-1429(2021).

    [90] O. Ronneberger, P. Fischer, T. Brox. U-Net: convolutional networks for biomedical image segmentation. Lect. Notes Comput. Sci., 9351, 234-241(2015).

    [91] X. Hua et al. Ultra-compact snapshot spectral light-field imaging. Nat. Commun., 13, 2732(2022).

    [92] J. Li et al. Single-layer Bayer metasurface via inverse design. ACS Photonics, 9, 2607-2613(2022).

    [93] X. Zou et al. Pixel-level Bayer-type colour router based on metasurfaces. Nat. Commun., 13, 3288(2022).

    [94] Z. Lin et al. End-to-end metasurface inverse design for single-shot multi-channel imaging. Opt. Express, 30, 28358-28370(2022).

    [95] E. Tseng et al. Neural nano-optics for high-quality thin lens imaging. Nat. Commun., 12, 6493(2021).

    [96] R. J. Lin et al. Achromatic metalens array for full-colour light-field imaging. Nat. Nanotechnol., 14, 227-231(2019).

    [97] S.-H. Baek et al. Compact single-shot hyperspectral imaging using a prism. ACM Trans. Graphics, 36, 217(2017).

    [98] B. H. Chen et al. GAN metalens for pixel-level full-color routing at visible light. Nano Lett., 17, 6345-6352(2017).

    [99] M. Chen et al. Full-color nanorouter for high-resolution imaging. Nanoscale, 13, 13024-13029(2021).

    [100] Z. Jin et al. Complex inverse design of metaoptics by segmented hierarchical evolutionary algorithm. ACS Nano, 13, 821-829(2019).

    [101] G. Strang. Computational science and engineering. Optimization, 551, 571-586(2007).

    [102] C. Li et al. An efficient augmented Lagrangian method with applications to total variation minimization. Comput. Optim. Appl., 56, 507-530(2013).

    [103] J. Yang, Y. Zhang. Alternating direction algorithms for l1-problems in compressive sensing. SIAM J. Sci. Comput., 33, 250-278(2011).

    [104] S. Zhuo, T. Sim. Defocus map estimation from a single image. Pattern Recognit., 44, 1852-1858(2011).

    [105] J. Salvi et al. A state-of-the-art in structured light patterns for surface profilometry. Pattern Recognit., 43, 2666-2680(2010).

    [106] J. Geng. Structured-light 3D surface imaging: a tutorial. Adv. Opt. Photonics, 3, 128-160(2011).

    [107] Y. Wu, H. Shroff. Faster, sharper, and deeper: structured illumination microscopy for biological imaging. Nat. Methods, 15, 1011-1019(2018).

    [108] IMEC develops snapshot hyperspectral image sensors(2015).

    [109] X. Hu et al. Convolutional sparse coding for RGB + NIR imaging. IEEE Trans. Image Process., 27, 1611-1625(2017).

    [110] Polarsens-polarization image sensors.

    [111] B. K. Gunturk et al. Demosaicking: color filter array interpolation. IEEE Signal Process Mag., 22, 44-54(2005).

    [112] S. Balakrishnama, A. Ganapathiraju. Linear discriminant analysis: a brief tutorial. Inst. Signal Inf. Process., 18, 1-8(1998).

    [113] L. K. Saul, S. T. Roweis. An introduction to locally linear embedding(2000).

    [114] D. Bank, L. Rokach, O. Maimon, N. Koenigstein, E. Shmueli, R. Giryes. Autoencoders. Machine Learning for Data Science Handbook: Data Mining and Knowledge Discovery Handbook, 353-374(2023).

    [115] M. S. Asif et al. FlatCam: thin, bare-sensor cameras using coded aperture and computation. IEEE Trans. Comput. Imaging, 3, 384-397(2017).

    [116] J. K. Adams et al. Single-frame 3D fluorescence microscopy with ultraminiature lensless FlatScope. Sci. Adv., 3, e1701548(2017).

    [117] A. Ozcan, E. McLeod. Lensless imaging and sensing. Ann. Rev. Biomed. Eng., 18, 77-102(2016).

    [118] V. Boominathan et al. Recent advances in lensless imaging. Optica, 9, 1-16(2022).

    [119] J. W. Goodman. Introduction to Fourier Optics(2005).

    [120] N. Yu et al. Light propagation with phase discontinuities: generalized laws of reflection and refraction. Science, 334, 333-337(2011).

    [121] N. Yu, F. Capasso. Flat optics with designer metasurfaces. Nat. Mater., 13, 139-150(2014).

    [122] S. So et al. Revisiting the design strategies for metasurfaces: fundamental physics, optimization, and beyond. Adv. Mater., 35, 2206399(2022).

    [123] Z. Lin et al. End-to-end nanophotonic inverse design for imaging and polarimetry. Nanophotonics, 10, 1177-1187(2020).

    [124] M. Mansouree et al. Multifunctional 2.5D metastructures enabled by adjoint optimization. Optica, 7, 77-84(2020).

    [125] J.-M. Jin. The Finite Element Method in Electromagnetics(2015).

    [126] A. F. Oskooi et al. Meep: a flexible free-software package for electromagnetic simulations by the FDTD method. Comput. Phys. Commun., 181, 687-702(2010).

    [127] W. Shin, S. Fan. Choice of the perfectly matched layer boundary condition for frequency-domain Maxwell’s equations solvers. J. Comput. Phys., 231, 3406-3431(2012).

    [128] T. J. R. Hughes. The Finite Element Method: Linear Static and Dynamic Finite Element Analysis(2012).

    [129] M. Makarenko et al. Robust and scalable flat-optics on flexible substrates via evolutionary neural networks. Advanced Intelligent Systems, 3, 2100105(2021).

    [130] H.-Y. Li et al. Intelligent electromagnetic sensing with learnable data acquisition and processing. Patterns, 1, 100006(2020).

    [131] L. Zhang. Fast stereo matching algorithm for intermediate view reconstruction of stereoscopic television images. IEEE Trans. Circuits Syst. Video Technol., 16, 1259-1270(2006).

    [132] L. I. Rudin, S. Osher, E. Fatemi. Nonlinear total variation-based noise removal algorithms. Physica D, 60, 259-268(1992).

    [133] N. Ahmed, T. Natarajan, K. R. Rao. Discrete cosine transform. IEEE Trans. Comput., C-23, 90-93(1974).

    [134] S. G. Mallat. A theory for multiresolution signal decomposition: the wavelet representation. IEEE Trans. Pattern Anal. Mach. Intell., 11, 674-693(1989).

    [135] H. Lee et al. Efficient sparse coding algorithms(2006).

    [136] X. Lin et al. Spatial-spectral encoded compressive hyperspectral imaging. ACM Trans. Graph., 33, 233(2014).

    [137] C. R. Gonzalez, R. E. Woods. Digital Image Processing(2002).

    [138] A. Beck, M. Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imag. Sci., 2, 183-202(2009).

    [139] Z. Wang et al. Uformer: a general U-shaped transformer for image restoration, 17683-17693(2022).

    [140] Z. Lu et al. Transformer for single image super-resolution, 457-466(2022).

    [141] A. Chakrabarti, T. Zickler. Statistics of real-world hyperspectral images, 193-200(2011).

    [142] F. Yasuma et al. Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum. IEEE Trans. Image Process., 19, 2241-2253(2010).

    [143] I. Choi et al. High-quality hyperspectral reconstruction using a spectral prior. ACM Trans. Graph., 36, 218(2017).

    [144] D. H. Foster et al. Frequency of metamerism in natural scenes. J. Opt. Soc. Am. A, 23, 2359-2372(2006).

    [145] S. Qiu et al. Polarization demosaicking for monochrome and color polarization focal plane arrays(2019).

    [146] Y. Ba et al. Deep shape from polarization. Lect. Notes Comput. Sci., 1269, 554-571(2020).

    [147] R. Blin et al. A new multimodal RGB and polarimetric image dataset for road scenes analysis, 216-217(2020).

    [148] A. Fan et al. Full-Stokes polarization multispectral images of various stereoscopic objects. Sci. Data, 10, 328(2023).

    [149] H. B. Barlow. Unsupervised learning. Neural Comput., 1, 295-311(1989).

    [150] Y. Wang et al. Generalizing from a few examples: a survey on few-shot learning. ACM Comput. Surv., 53, 63(2020).

    [151] S. Hochreiter. The vanishing gradient problem during learning recurrent neural nets and problem solutions. Int. J. Uncertain. Fuzziness Knowl.-Based Syst., 6, 107-116(1998).

    [152] Z. Jia, H. Su. Information-theoretic local minima characterization and regularization, 4773-4783(2020).

    [153] S. Ioffe, C. Szegedy. Batch normalization: accelerating deep network training by reducing internal covariate shift, 448-456(2015).

    [154] Y. Wu et al. Phasecam3D—learning phase masks for passive single view depth estimation, 1-12(2019).

    Xuemei Hu, Weizhu Xu, Qingbin Fan, Tao Yue, Feng Yan, Yanqing Lu, Ting Xu. Metasurface-based computational imaging: a review[J]. Advanced Photonics, 2024, 6(1): 014002
    Download Citation