• Photonics Research
  • Vol. 9, Issue 12, 2464 (2021)
Xianglei Liu1、†, João Monteiro1、†, Isabela Albuquerque1, Yingming Lai1, Cheng Jiang1, Shian Zhang2, Tiago H. Falk1、3、*, and Jinyang Liang1、4、*
Author Affiliations
  • 1Centre Énergie Matériaux Télécommunications, Institut National de la Recherche Scientifique, Varennes, Québec J3X1S2, Canada
  • 2State Key Laboratory of Precision Spectroscopy, East China Normal University, Shanghai 200062, China
  • 3e-mail: falk@emt.inrs.ca
  • 4e-mail: jinyang.liang@emt.inrs.ca
  • show less
    DOI: 10.1364/PRJ.422179 Cite this Article Set citation alerts
    Xianglei Liu, João Monteiro, Isabela Albuquerque, Yingming Lai, Cheng Jiang, Shian Zhang, Tiago H. Falk, Jinyang Liang. Single-shot real-time compressed ultrahigh-speed imaging enabled by a snapshot-to-video autoencoder[J]. Photonics Research, 2021, 9(12): 2464 Copy Citation Text show less
    References

    [1] M. Kannan, G. Vasan, C. Huang, S. Haziza, J. Z. Li, H. Inan, M. J. Schnitzer, V. A. Pieribone. Fast, in vivo voltage imaging using a red fluorescent indicator. Nat. Methods, 15, 1108-1116(2018).

    [2] M. Sasaki, A. Matsunaka, T. Inoue, K. Nishio, Y. Awatsuji. Motion-picture recording of ultrafast behavior of polarized light incident at Brewster’s angle. Sci. Rep., 10, 7638(2020).

    [3] P. R. Poulin, K. A. Nelson. Irreversible organic crystalline chemistry monitored in real time. Science, 313, 1756-1760(2006).

    [4] K. Toru, T. Yoshiaki, K. Kenji, T. Mitsuhiro, T. Naohiro, K. Hideki, S. Shunsuke, A. Jun, S. Haruhisa, G. Yuichi, M. Seisuke, T. Yoshitaka. A 3D stacked CMOS image sensor with 16 Mpixel global-shutter mode and 2 Mpixel 10000  fps mode using 4 million interconnections. IEEE Symposium on VLSI Circuits, C90-C91(2015).

    [5] T. Etoh, V. Dao, K. Shimonomura, E. Charbon, C. Zhang, Y. Kamakura, T. Matsuoka. Toward 1Gfps: evolution of ultra-high-speed image sensors-ISIS, BSI, multi-collection gates, and 3D-stacking. IEEE IEDM, 11-14(2014).

    [6] T. York, S. B. Powell, S. Gao, L. Kahan, T. Charanya, D. Saha, N. Roberts, T. Cronin, N. Marshall, S. Achilefu, S. Lake, B. Raman, V. Gruev. Bioinspired polarization imaging sensors: from circuits and optics to signal processing algorithms and biomedical applications. Proc. IEEE, 102, 1450-1469(2014).

    [7] D. Calvet. A new interface technique for the acquisition of multiple multi-channel high speed ADCs. IEEE Trans. Nucl. Sci., 55, 2592-2597(2008).

    [8] M. Hejtmánek, G. Neue, P. Voleš. Software interface for high-speed readout of particle detectors based on the CoaXPress communication standard. J. Instrum., 10, C06011(2015).

    [9] G. Barbastathis, A. Ozcan, G. Situ. On the use of deep learning for computational imaging. Optica, 6, 921-943(2019).

    [10] A. Ehn, J. Bood, Z. Li, E. Berrocal, M. Aldén, E. Kristensson. FRAME: femtosecond videography for atomic and molecular dynamics. Light Sci. Appl., 6, e17045(2017).

    [11] Z. Li, R. Zgadzaj, X. Wang, Y.-Y. Chang, M. C. Downer. Single-shot tomographic movies of evolving light-velocity objects. Nat. Commun., 5, 3085(2014).

    [12] D. Qi, S. Zhang, C. Yang, Y. He, F. Cao, J. Yao, P. Ding, L. Gao, T. Jia, J. Liang, Z. Sun, L. V. Wang. Single-shot compressed ultrafast photography: a review. Adv. Photon., 2, 014003(2020).

    [13] P. Wang, J. Liang, L. V. Wang. Single-shot ultrafast imaging attaining 70 trillion frames per second. Nat. Commun., 11, 2091(2020).

    [14] J. Liang, L. Zhu, L. V. Wang. Single-shot real-time femtosecond imaging of temporal focusing. Light Sci. Appl., 7, 42(2018).

    [15] Y. Lai, Y. Xue, C. Y. Côté, X. Liu, A. Laramée, N. Jaouen, F. Légaré, L. Tian, J. Liang. Single-shot ultraviolet compressed ultrafast photography. Laser Photon. Rev., 14, 2000122(2020).

    [16] J. Liang, P. Wang, L. Zhu, L. V. Wang. Single-shot stereo-polarimetric compressed ultrafast photography for light-speed observation of high-dimensional optical transients with picosecond resolution. Nat. Commun., 11, 5252(2020).

    [17] C. Yang, F. Cao, D. Qi, Y. He, P. Ding, J. Yao, T. Jia, Z. Sun, S. Zhang. Hyperspectrally compressed ultrafast photography. Phys. Rev. Lett., 124, 023902(2020).

    [18] J. Liang, C. Ma, L. Zhu, Y. Chen, L. Gao, L. V. Wang. Single-shot real-time video recording of a photonic Mach cone induced by a scattered light pulse. Sci. Adv., 3, e1601814(2017).

    [19] X. Liu, S. Zhang, A. Yurtsever, J. Liang. Single-shot real-time sub-nanosecond electron imaging aided by compressed sensing: analytical modeling and simulation. Micron, 117, 47-54(2019).

    [20] L. Gao, J. Liang, C. Li, L. V. Wang. Single-shot compressed ultrafast photography at one hundred billion frames per second. Nature, 516, 74-77(2014).

    [21] J. Liang, L. V. Wang. Single-shot ultrafast optical imaging. Optica, 5, 1113-1127(2018).

    [22] J. Liang. Punching holes in light: recent progress in single-shot coded-aperture optical imaging. Rep. Prog. Phys., 83, 116101(2020).

    [23] J. Yang, X. Yuan, X. Liao, P. Llull, D. J. Brady, G. Sapiro, L. Carin. Video compressive sensing using Gaussian mixture models. IEEE Trans. Image Process., 23, 4863-4878(2014).

    [24] C. Wang, Z. Cheng, W. Gan, M. Cui. Line scanning mechanical streak camera for phosphorescence lifetime imaging. Opt. Express, 28, 26717-26723(2020).

    [25] X. Liu, J. Liu, C. Jiang, F. Vetrone, J. Liang. Single-shot compressed optical-streaking ultra-high-speed photography. Opt. Lett., 44, 1387-1390(2019).

    [26] P. Llull, X. Liao, X. Yuan, J. Yang, D. Kittle, L. Carin, G. Sapiro, D. J. Brady. Coded aperture compressive temporal imaging. Opt. Express, 21, 10526-10545(2013).

    [27] R. Koller, L. Schmid, N. Matsuda, T. Niederberger, L. Spinoulas, O. Cossairt, G. Schuster, A. K. Katsaggelos. High spatio-temporal resolution video with compressed sensing. Opt. Express, 23, 15992-16007(2015).

    [28] D. Reddy, A. Veeraraghavan, R. Chellappa. P2C2: programmable pixel compressive camera for high speed imaging. IEEE CVPR, 329-336(2011).

    [29] Y. Liu, X. Yuan, J. Suo, D. J. Brady, Q. Dai. Rank minimization for snapshot compressive imaging. IEEE Trans. Pattern Anal. Mach. Intell., 41, 2990-3006(2018).

    [30] A. Lucas, M. Iliadis, R. Molina, A. K. Katsaggelos. Using deep neural networks for inverse problems in imaging beyond analytical methods. IEEE Signal Process. Mag., 35, 20-36(2018).

    [31] J. M. Bioucas-Dias, M. A. Figueiredo. A new TwIST: two-step iterative shrinkage/thresholding algorithms for image restoration. IEEE Trans. Image Process., 16, 2992-3004(2007).

    [32] C. Yang, D. Qi, F. Cao, Y. He, X. Wang, W. Wen, J. Tian, T. Jia, Z. Sun, S. Zhang. Improving the image reconstruction quality of compressed ultrafast photography via an augmented Lagrangian algorithm. J. Opt., 21, 035703(2019).

    [33] J. Hui, Y. Cao, Y. Zhang, A. Kole, P. Wang, G. Yu, G. Eakins, M. Sturek, W. Chen, J.-X. Cheng. Real-time intravascular photoacoustic-ultrasound imaging of lipid-laden plaque in human coronary artery at 16 frames per second. Sci. Rep., 7, 1417(2017).

    [34] M. Kreizer, D. Ratner, A. Liberzon. Real-time image processing for particle tracking velocimetry. Exp. Fluids, 48, 105-110(2010).

    [35] Y. LeCun, Y. Bengio, G. Hinton. Deep learning. Nature, 521, 436-444(2015).

    [36] M. Iliadis, L. Spinoulas, A. K. Katsaggelos. Deep fully-connected networks for video compressive sensing. Digit. Signal Process., 72, 9-18(2018).

    [37] M. Yoshida, A. Torii, M. Okutomi, K. Endo, Y. Sugiyama, R.-I. Taniguchi, H. Nagahara. Joint optimization for compressive video sensing and reconstruction under hardware constraints. Proceedings of the European Conference on Computer Vision (ECCV), 634-649(2018).

    [38] M. Qiao, Z. Meng, J. Ma, X. Yuan. Deep learning for video compressive sensing. APL Photon., 5, 030801(2020).

    [39] Y. Ma, X. Feng, L. Gao. Deep-learning-based image reconstruction for compressed ultrafast photography. Opt. Lett., 45, 4400-4403(2020).

    [40] C. Yang, Y. Yao, C. Jin, D. Qi, F. Cao, Y. He, J. Yao, P. Ding, L. Gao, T. Jia. High-fidelity image reconstruction for compressed ultrafast photography via an augmented-Lagrangian and deep-learning hybrid algorithm. Photon. Res., 9, B30-B37(2021).

    [41] A. Zhang, J. Wu, J. Suo, L. Fang, H. Qiao, D. D.-U. Li, S. Zhang, J. Fan, D. Qi, Q. Dai. Single-shot compressed ultrafast photography based on U-net network. Opt. Express, 28, 39299-39310(2020).

    [42] M. W. Gardner, S. Dorling. Artificial neural networks (the multilayer perceptron)—a review of applications in the atmospheric sciences. Atmos. Environ., 32, 2627-2636(1998).

    [43] O. Ronneberger, P. Fischer, T. Brox. U-net: Convolutional Networks for Biomedical Image Segmentation, 234-241(2015).

    [44] Z. Cheng, R. Lu, Z. Wang, H. Zhang, B. Chen, Z. Meng, X. Yuan. BIRNAT: bidirectional recurrent neural networks with adversarial training for video snapshot compressive imaging. ECCV, 258-275(2020).

    [45] M. Tschannen, O. Bachem, M. Lucic. Recent advances in autoencoder-based representation learning(2018).

    [46] A. Nguyen, J. Clune, Y. Bengio, A. Dosovitskiy, J. Yosinski. Plug & play generative networks: conditional iterative generation of images in latent space. IEEE CVPR, 4467-4477(2017).

    [47] A. B. L. Larsen, S. K. Sønderby, H. Larochelle, O. Winther. Autoencoding beyond pixels using a learned similarity metric. PMLR International Conference on Machine Learning, 1558-1566(2016).

    [48] C. Vondrick, H. Pirsiavash, A. Torralba. Generating videos with scene dynamics. Adv. Neural Inf. Process Syst., 29, 613-621(2016).

    [49] S. Tulyakov, M.-Y. Liu, X. Yang, J. Kautz. Mocogan: decomposing motion and content for video generation. IEEE CVPR, 1526-1535(2018).

    [50] K. Ohnishi, S. Yamamoto, Y. Ushiku, T. Harada. Hierarchical video generation from orthogonal information: optical flow and texture(2017).

    [51] O. Plchot, L. Burget, H. Aronowitz, P. Matejka. Audio enhancing with DNN autoencoder for speaker recognition. IEEE ICASSP, 5090-5094(2016).

    [52] J. Yu, X. Zheng, S. Wang. A deep autoencoder feature learning method for process pattern recognition. J. Process Control, 79, 1-15(2019).

    [53] M. A. Ranzato, C. Poultney, S. Chopra, Y. L. Cun. Efficient learning of sparse representations with an energy-based model. Advances in Neural Information Processing Systems, 1137-1144(2007).

    [54] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, P.-A. Manzagol, L. Bottou. Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res., 11, 3371-3408(2010).

    [55] J. Liang, M. F. Becker, R. N. Kohn, D. J. Heinzen. Homogeneous one-dimensional optical lattice generation using a digital micromirror device-based high-precision beam shaper. J. Micro/Nanolithogr. MEMS MOEMS, 11, 023002(2012).

    [56] X. Ma, E. Hovy. End-to-end sequence labeling via bi-directional LSTM-CNNS-CRF(2016).

    [57] S. Ioffe, C. Szegedy. Batch normalization: accelerating deep network training by reducing internal covariate shift. Proceedings of the 32nd International Conference on Machine Learning, 448-456(2015).

    [58] V. Nair, G. E. Hinton. Rectified linear units improve restricted boltzmann machines. Proceedings of the 27th International Conference on Machine Learning, 807-814(2010).

    [59] Z. Zhang, M. Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in Neural Information Processing Systems, 8778-8788(2018).

    [60] A. Krogh, J. A. Hertz. A simple weight decay can improve generalization. Advances in Neural Information Processing Systems, 950-957(1992).

    [61] D. P. Kingma, J. Ba. Adam: a method for stochastic optimization(2014).

    [62] L. Deng. The MNIST database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Process. Mag., 29, 141-142(2012).

    [63] Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process., 13, 600-612(2004).

    [64] . Register images using registration estimator app.

    [65] C. Jiang, P. Kilcullen, Y. Lai, T. Ozaki, J. Liang. High-speed dual-view band-limited illumination profilometry using temporally interlaced acquisition. Photon. Res., 8, 1808-1817(2020).

    [66] X. Yuan, Y. Liu, J. Suo, Q. Dai. Plug-and-play algorithms for large-scale snapshot compressive imaging. CVPR, 1447-1457(2020).

    [67] Z. Lin, A. Khetan, G. Fanti, S. Oh. PACGAN: the power of two samples in generative adversarial networks. Advances in Neural Information Processing Systems, 1498-1507(2018).

    [68] A. Jolicoeur-Martineau. The relativistic discriminator: a key element missing from standard GAN(2018).

    [69] B. Neyshabur, S. Bhojanapalli, A. Chakrabarti. Stabilizing GAN training with multiple random projections(2017).

    [70] I. Albuquerque, J. Monteiro, T. Doan, B. Considine, T. Falk, I. Mitliagkas. Multi-objective training of generative adversarial networks with multiple discriminators. Proceedings of the 36th International Conference on Machine Learning, 202-211(2019).

    [71] P. Razvan, T. Mikolov, Y. Bengio. On the difficulty of training recurrent neural networks. Proceedings of the 30th International Conference on Machine Learning, 1310-1318(2013).

    [72] P. Ding, Y. Yao, D. Qi, C. Yang, F. Cao, Y. He, J. Yao, C. Jin, Z. Huang, L. Deng, L. Deng, T. Jia, J. Liang, Z. Sun, S. Zhang. Single-shot spectral-volumetric compressed ultrafast photography. Adv. Photon., 3, 045001(2021).

    [73] Z. Meng, X. Yuan. Perception inspired deep neural networks for spectral snapshot compressive imaging. ICIP, 2813-2817(2021).

    [74] Y. Pu, Z. Gan, R. Henao, X. Yuan, C. Li, A. Stevens, L. Carin. Variational autoencoder for deep learning of images, labels and captions. Advances in Neural Information Processing Systems, 2352-2360(2016).

    [75] A. Ten Cate, C. H. Nieuwstad, J. J. Derksen, H. E. A. Van den Akker. Particle imaging velocimetry experiments and lattice-Boltzmann simulations on a single sphere settling under gravity. Phys. Fluids, 14, 4012-4025(2002).

    [76] N. Nitta, T. Sugimura, A. Isozaki, H. Mikami, K. Hiraki, S. Sakuma, T. Iino, F. Arai, T. Endo, Y. Fujiwaki, H. Fukuzawa, M. Hase, T. Hayakawa, K. Hiramatsu, Y. Hoshino, M. Inaba, T. Ito, H. Karakawa, Y. Kasai, K. Koizumi, S. Lee, C. Lei, M. Li, T. Maeno, S. Matsusaka, D. Murakami, A. Nakagawa, Y. Oguchi, M. Oikawa, T. Ota, K. Shiba, H. Shintaku, Y. Shirasaki, K. Suga, Y. Suzuki, N. Suzuki, Y. Tanaka, H. Tezuka, C. Toyokawa, Y. Yalikun, M. Yamada, M. Yamagishi, T. Yamano, A. Yasumoto, Y. Yatomi, M. Yazawa, D. Di Carlo, Y. Hosokawa, S. Uemura, Y. Ozeki, K. Goda. Intelligent image-activated cell sorting. Cell, 175, 266-276(2018).

    Xianglei Liu, João Monteiro, Isabela Albuquerque, Yingming Lai, Cheng Jiang, Shian Zhang, Tiago H. Falk, Jinyang Liang. Single-shot real-time compressed ultrahigh-speed imaging enabled by a snapshot-to-video autoencoder[J]. Photonics Research, 2021, 9(12): 2464
    Download Citation