• Photonics Research
  • Vol. 9, Issue 2, B18 (2021)
Siming Zheng1、2、†, Yang Liu3、†, Ziyi Meng4, Mu Qiao5, Zhishen Tong6、7, Xiaoyu Yang1、2, Shensheng Han6、7, and Xin Yuan8、*
Author Affiliations
  • 1Computer Network Information Center, Chinese Academy of Sciences, Beijing 100190, China
  • 2University of Chinese Academy of Sciences, Beijing 100049, China
  • 3Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139, USA
  • 4Beijing University of Posts and Telecommunications, Beijing 100876, China
  • 5New Jersey Institute of Technology, Newark, New Jersey 07102, USA
  • 6Key Laboratory for Quantum Optics and Center for Cold Atom Physics of CAS, Shanghai Institute of Optics and Fine Mechanics, Chinese Academy of Sciences, Shanghai 201800, China
  • 7Center of Materials Science and Optoelectronics Engineering, University of Chinese Academy of Sciences, Beijing 100049, China
  • 8Nokia Bell Labs, Murray Hill, New Jersey 07974, USA
  • show less
    DOI: 10.1364/PRJ.411745 Cite this Article Set citation alerts
    Siming Zheng, Yang Liu, Ziyi Meng, Mu Qiao, Zhishen Tong, Xiaoyu Yang, Shensheng Han, Xin Yuan. Deep plug-and-play priors for spectral snapshot compressive imaging[J]. Photonics Research, 2021, 9(2): B18 Copy Citation Text show less
    References

    [1] B. E. Bayer. Color imaging array. U.S. patent(1976).

    [2] B. Redding, S. F. Liew, R. Sarma, H. Cao. Compact spectrometer based on a disordered photonic chip. Nat. Photonics, 7, 746-751(2013).

    [3] Z. Wang, Z. Yu. Spectral analysis based on compressive sensing in nanophotonic structures. Opt. Express, 22, 25608-25614(2014).

    [4] J. Bao, M. G. Bawendi. A colloidal quantum dot spectrometer. Nature, 523, 67-70(2015).

    [5] Z. Yang, T. Albrow-Owen, H. Cui, J. Alexander-Webber, F. Gu, X. Wang, T.-C. Wu, M. Zhuge, C. Williams, P. Wang, V. A. Zayats, W. Cai, L. Dai, S. Hofmann, M. Overend, L. Tong, Q. Yang, Z. Sun, T. Hasan. Single-nanowire spectrometers. Science, 365, 1017-1020(2019).

    [6] Z. Wang, S. Yi, A. Chen, M. Zhou, T. S. Luk, A. James, J. Nogan, W. Ross, G. Joe, A. Shahsafi, K. X. Wang, M. A. Kats, Z. Yu. Single-shot on-chip spectral sensors based on photonic crystal slabs. Nat. Commun., 10, 1020(2019).

    [7] A. McClung, S. Samudrala, M. Torfeh, M. Mansouree, A. Arbabi. Snapshot spectral imaging with parallel metasystems. Sci. Adv., 6, eabc7646(2020).

    [8] Y. Kwak, S. M. Park, Z. Ku, A. Urbas, Y. L. Kim. A pearl spectrometer. Nano Lett.(2020).

    [9] D. Donoho. Compressed sensing. IEEE Trans. Inf. Theory, 52, 1289-1306(2006).

    [10] E. J. Candès, J. Romberg, T. Tao. Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory, 52, 489-509(2006).

    [11] E. J. Candès, M. B. Wakin. An introduction to compressive sampling. IEEE Signal Process. Mag., 25, 21-30(2008).

    [12] M. E. Gehm, R. John, D. J. Brady, R. M. Willett, T. J. Schulz. Single-shot compressive spectral imaging with a dual-disperser architecture. Opt. Express, 15, 14013-14027(2007).

    [13] G. R. Arce, D. J. Brady, L. Carin, H. Arguello, D. S. Kittle. Compressive coded aperture spectral imaging: an introduction. IEEE Signal Process. Mag., 31, 105-115(2014).

    [14] X. Cao, T. Yue, X. Lin, S. Lin, X. Yuan, Q. Dai, L. Carin, D. J. Brady. Computational snapshot multispectral cameras: toward dynamic capture of the spectral world. IEEE Signal Process. Mag., 33, 95-108(2016).

    [15] J. Bioucas-Dias, M. Figueiredo. A new TwIST: two-step iterative shrinkage/thresholding algorithms for image restoration. IEEE Trans. Image Process., 16, 2992-3004(2007).

    [16] Y. Liu, X. Yuan, J. Suo, D. J. Brady, Q. Dai. Rank minimization for snapshot compressive imaging. IEEE Trans. Pattern Anal. Mach. Intell., 41, 2990-3006(2019).

    [17] L. Wang, Z. Xiong, G. Shi, F. Wu, W. Zeng. Adaptive nonlocal sparse representation for dual-camera compressive hyperspectral imaging. IEEE Trans. Pattern Anal. Mach. Intell., 39, 2104-2111(2017).

    [18] X. Yuan, Y. Liu, J. Suo, Q. Dai. Plug-and-play algorithms for large-scale snapshot compressive imaging. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 1447-1457(2020).

    [19] A. Wagadarikar, R. John, R. Willett, D. J. Brady. Single disperser design for coded aperture snapshot spectral imaging. Appl. Opt., 47, B44-B51(2008).

    [20] X. Lin, Y. Liu, J. Wu, Q. Dai. Spatial-spectral encoded compressive hyperspectral imaging. ACM Trans. Graph., 33, 233(2014).

    [21] X. Cao, H. Du, X. Tong, Q. Dai, S. Lin. A prism-mask system for multispectral video acquisition. IEEE Trans. Pattern Anal. Mach. Intell., 33, 2423-2435(2011).

    [22] H. Arguello, H. Rueda, Y. Wu, D. W. Prather, G. R. Arce. Higher-order computational model for coded aperture spectral imaging. Appl. Opt., 52, D12-D21(2013).

    [23] L. Wang, Z. Xiong, D. Gao, G. Shi, F. Wu. Dual-camera design for coded aperture snapshot spectral imaging. Appl. Opt., 54, 848-858(2015).

    [24] C. V. Correa, H. Arguello, G. R. Arce. Snapshot colored compressive spectral imager. J. Opt. Soc. Am. A, 32, 1754-1763(2015).

    [25] X. Yuan, T.-H. Tsai, R. Zhu, P. Llull, D. J. Brady, L. Carin. Compressive hyperspectral imaging with side information. IEEE J. Sel. Top. Signal Process., 9, 964-976(2015).

    [26] Z. Liu, S. Tan, J. Wu, E. Li, X. Shen, S. Han. Spectral camera based on ghost imaging via sparsity constraints. Sci. Rep., 6, 25718(2016).

    [27] X. Li, J. A. Greenberg, M. E. Gehm. Single-shot multispectral imaging through a thin scatterer. Optica, 6, 864-871(2019).

    [28] D. S. Jeon, S.-H. Baek, S. Yi, Q. Fu, X. Dun, W. Heidrich, M. H. Kim. Compact snapshot hyperspectral imaging with diffracted rotation. ACM Trans. Graph., 38, 117(2019).

    [29] S.-H. Baek, I. Kim, D. Gutierrez, M. H. Kim. Compact single-shot hyperspectral imaging using a prism. ACM Trans. Graph., 36(2017).

    [30] M. A. T. Figueiredo, R. D. Nowak, S. J. Wright. Gradient projection for sparse reconstruction: application to compressed sensing and other inverse problems. IEEE J. Sel. Top. Signal Process., 1, 586-597(2007).

    [31] M. Aharon, M. Elad, A. Bruckstein. K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans. Signal Process., 54, 4311-4322(2006).

    [32] J. Yang, X. Liao, X. Yuan, P. Llull, D. J. Brady, G. Sapiro, L. Carin. Compressive sensing by learning a Gaussian mixture model from measurements. IEEE Trans. Image Process., 24, 106-119(2015).

    [33] G. Barbastathis, A. Ozcan, G. Situ. On the use of deep learning for computational imaging. Optica, 6, 921-943(2019).

    [34] X. Yuan, Y. Pu. Parallel lensless compressive imaging via deep convolutional neural networks. Opt. Express, 26, 1962-1977(2018).

    [35] I. Choi, D. S. Jeon, G. Nam, D. Gutierrez, M. H. Kim. High-quality hyperspectral reconstruction using a spectral prior. ACM Trans. Graph., 36, 218(2017).

    [36] Z. Meng, M. Qiao, J. Ma, Z. Yu, K. Xu, X. Yuan. Snapshot multispectral endomicroscopy. Opt. Lett., 45, 3897-3900(2020).

    [37] X. Miao, X. Yuan, Y. Pu, V. Athitsos. λ-net: reconstruct hyperspectral images from a snapshot measurement. IEEE/CVF Conference on Computer Vision (ICCV), 4058-4068(2019).

    [38] L. Wang, C. Sun, Y. Fu, M. H. Kim, H. Huang. Hyperspectral image reconstruction using a deep spatial-spectral prior. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 8024-8033(2019).

    [39] Z. Meng, J. Ma, X. Yuan. End-to-end low cost compressive spectral imaging with spatial-spectral self-attention. European Conference on Computer Vision (ECCV), 187-204(2020).

    [40] S. V. Venkatakrishnan, C. A. Bouman, B. Wohlberg. Plug-and-play priors for model based reconstruction. IEEE Global Conference on Signal and Information Processing, 945-948(2013).

    [41] S. Sreehari, S. V. Venkatakrishnan, B. Wohlberg, G. T. Buzzard, L. F. Drummy, J. P. Simmons, C. A. Bouman. Plug-and-play priors for bright field electron tomography and sparse interpolation. IEEE Trans. Comput. Imaging, 2, 408-423(2016).

    [42] S. H. Chan, X. Wang, O. A. Elgendy. Plug-and-play ADMM for image restoration: fixed-point convergence and applications. IEEE Trans. Comput. Imaging, 3, 84-98(2017).

    [43] E. K. Ryu, J. Liu, S. Wang, X. Chen, Z. Wang, W. Yin. Plug-and-play methods provably converge with properly trained denoisers(2019).

    [44] L. Zhang, W. Zuo. Image restoration: from sparse and low-rank priors to deep priors. IEEE Signal Process. Mag., 34, 172-179(2017).

    [45] K. Zhang, W. Zuo, Y. Chen, D. Meng, L. Zhang. Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans. Image Process., 26, 3142-3155(2017).

    [46] K. Zhang, W. Zuo, L. Zhang. FFDNet: toward a fast and flexible solution for CNN-based image denoising. IEEE Trans. Image Process., 27, 4608-4622(2018).

    [47] A. A. Wagadarikar, N. P. Pitsianis, X. Sun, D. J. Brady. Video rate spectral imaging using a coded aperture snapshot spectral imager. Opt. Express, 17, 6368-6388(2009).

    [48] D. Kittle, K. Choi, A. Wagadarikar, D. J. Brady. Multiframe image estimation for coded aperture snapshot spectral imagers. Appl. Opt., 49, 6824-6833(2010).

    [49] R. Zhu, T. Tsai, D. J. Brady. Coded aperture snapshot spectral imager based on liquid crystal spatial light modulator. Frontiers in Optics, FW1D-4(2013).

    [50] T.-H. Tsai, X. Yuan, D. J. Brady. Spatial light modulator based color polarization imaging. Opt. Express, 23, 11912-11926(2015).

    [51] X. Yuan. Generalized alternating projection based total variation minimization for compressive sensing. IEEE International Conference on Image Processing (ICIP), 2539-2543(2016).

    [52] S. K. Sahoo, D. Tang, C. Dang. Single-shot multispectral imaging with a monochromatic camera. Optica, 4, 1209-1213(2017).

    [53] K. Monakhova, K. Yanny, N. Aggarwal, L. Waller. Spectral diffusercam: lensless snapshot hyperspectral imaging with a spectral filter array. Optica, 7, 1298-1307(2020).

    [54] J. Wu, E. Li, X. Shen, S. Yao, Z. Tong, C. Hu, Z. Liu, S. Liu, S. Tan, S. Han. Experimental results of the balloon-borne spectral camera based on ghost imaging via sparsity constraints. IEEE Access, 6, 68740-68748(2018).

    [55] E. Candès, J. Romberg, T. Tao. Stable signal recovery from incomplete and inaccurate measurements. Commun. Pure Appl. Math., 59, 1207-1223(2006).

    [56] E. J. Candès, T. Tao. Near-optimal signal recovery from random projections: universal encoding strategies?. IEEE Trans. Inf. Theory, 52, 5406-5425(2006).

    [57] E. J. Candès. The restricted isometry property and its implications for compressed sensing. C.R. Math., 346, 589-592(2008).

    [58] S. Jalali, X. Yuan. Snapshot compressed sensing: performance bounds and algorithms. IEEE Trans. Inf. Theory, 65, 8005-8024(2019).

    [59] S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn., 3, 1-122(2011).

    [60] N. Parikh, S. Boyd. Proximal algorithms. Found. Trends Optim., 1, 127-239(2014).

    [61] W. W. Hager. Updating the inverse of a matrix. SIAM Rev., 31, 221-239(1989).

    [62] I. Daubechies, M. Defrise, C. De Mol. An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun. Pure Appl. Math., 57, 1413-1457(2004).

    [63] M. Gharbi, G. Chaurasia, S. Paris, F. Durand. Deep joint demosaicking and denoising. ACM Trans. Graph., 35(2016).

    [64] M. Tassano, J. Delon, T. Veit. FastDVDnet: towards real-time deep video denoising without flow estimation. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 1354-1363(2020).

    [65] A. Maffei, J. M. Haut, M. E. Paoletti, J. Plaza, L. Bruzzone, A. Plaza. A single model CNN for hyperspectral image denoising. IEEE Trans. Geosci. Remote Sens., 58, 2516-2529(2020).

    [66] F. Yasuma, T. Mitsunaga, D. Iso, S. K. Nayar. Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum. IEEE Trans. Image Process., 19, 2241-2253(2010).

    [67] H. Wallach, A. Paszke, S. Gross, H. Larochelle, F. Massa, A. Beygelzimer, A. Lerer, F. d’ Alché-Buc, J. Bradbury, E. Fox, R. Garnett, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, S. Chintala. PyTorch: an imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems 32, 8024-8035(2019).

    [68] D. P. Kingma, J. Ba. Adam: a method for stochastic optimization(2014).

    [69] B. Arad, O. Ben-Shahar. Sparse recovery of hyperspectral signal from natural RGB images. European Conference on Computer Vision, 19-34(2016).

    [70] O. Ronneberger, P. Fischer, T. Brox. U-net: convolutional networks for biomedical image segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention, 234-241(2015).

    [71] Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process., 13, 600-612(2004).

    [72] T. Smith, J. Guild. The C.I.E. colorimetric standards and their use. Trans. Opt. Soc., 33, 73-134(1931).

    CLP Journals

    [1] Lishun Wang, Zongliang Wu, Yong Zhong, Xin Yuan. Snapshot spectral compressive imaging reconstruction using convolution and contextual Transformer[J]. Photonics Research, 2022, 10(8): 1848

    [2] Li Gao, Yang Chai, Darko Zibar, Zongfu Yu. Deep learning in photonics: introduction[J]. Photonics Research, 2021, 9(8): DLP1

    Siming Zheng, Yang Liu, Ziyi Meng, Mu Qiao, Zhishen Tong, Xiaoyu Yang, Shensheng Han, Xin Yuan. Deep plug-and-play priors for spectral snapshot compressive imaging[J]. Photonics Research, 2021, 9(2): B18
    Download Citation