• Acta Optica Sinica
  • Vol. 40, Issue 1, 0111003 (2020)
Chao Zuo1、2, Shijie Feng1、2, Xiangyu Zhang1、2, Jing Han2, and Chen Qian2、*
Author Affiliations
  • 1Smart Computational Imaging Laboratory (SCILab), School of Electronic and Optical Engineering, Nanjing University of Science and Technology, Nanjing, Jiangsu 210094, China
  • 2Jiangsu Key Laboratory of Spectral Imaging & Intelligent Sense, Nanjing University of Science and Technology, Nanjing, Jiangsu 210094, China;
  • show less
    DOI: 10.3788/AOS202040.0111003 Cite this Article Set citation alerts
    Chao Zuo, Shijie Feng, Xiangyu Zhang, Jing Han, Chen Qian. Deep Learning Based Computational Imaging: Status, Challenges, and Future[J]. Acta Optica Sinica, 2020, 40(1): 0111003 Copy Citation Text show less
    References

    [1] Wikipedia. Computational imaging[EB/OL]. -10-15)[2019-11-20](2019). https://en.wikipedia.org/wiki/Computational_imaging.

    [3] Russakovsky O, Deng J, Su H et al. Imagenet large scale visual recognition challenge[J]. International Journal of Computer Vision, 115, 211-252(2015).

    [4] Rivenson Y, Zhang Y B, Günaydın H et al. Phase recovery and holographic image reconstruction using deep learning in neural networks[J]. Light: Science & Applications, 7, 17141(2018).

    [5] Wu Y C, Rivenson Y, Zhang Y B et al. Extended depth-of-field in holographic imaging using deep-learning-based autofocusing and phase recovery[J]. Optica, 5, 704-710(2018).

    [6] Ren Z B, Xu Z M, Lam E Y. Learning-based nonparametric autofocusing for digital holography[J]. Optica, 5, 337-344(2018).

    [7] Wang K Q, Dou J Z, Qian K M et al. Y-Net: a one-to-two deep learning framework for digital holographic reconstruction[J]. Optics Letters, 44, 4765-4768(2019).

    [8] Nguyen T, Bui V, Lam V et al. Automatic phase aberration compensation for digital holographic microscopy based on deep learning background detection[J]. Optics Express, 25, 15043-15057(2017).

    [9] Zhang G, Guan T, Shen Z Y et al. Fast phase retrieval in off-axis digital holographic microscopy through deep learning[J]. Optics Express, 26, 19388-19405(2018).

    [10] Nguyen T, Xue Y J, Li Y Z et al. Deep learning approach for Fourier ptychography microscopy[J]. Optics Express, 26, 26470-26484(2018).

    [11] Kappeler A, Ghosh S, Holloway J et al. Ptychnet: CNN based Fourier ptychography. [C]∥2017 IEEE International Conference on Image Processing (ICIP), September 17-20, 2017, Beijing, China. New York: IEEE, 1712-1716(2017).

    [12] Jiang S W, Guo K K, Liao J et al. Solving Fourier ptychographic imaging problems via neural network modeling and TensorFlow[J]. Biomedical Optics Express, 9, 3306-3319(2018).

    [13] Cheng Y F, Strachan M, Weiss Z et al. Illumination pattern design with deep learning for single-shot Fourier ptychographic microscopy[J]. Optics Express, 27, 644-656(2019).

    [14] Lü M, Wang W, Wang H et al. Deep-learning-based ghost imaging[J]. Scientific Reports, 7, 17865(2017).

    [15] He Y C, Wang G, Dong G X et al. Ghost imaging based on deep learning[J]. Scientific Reports, 8, 6469(2018).

    [16] Shimobaba T, Endo Y, Nishitsuji T et al. Computational ghost imaging using deep learning[J]. Optics Communications, 413, 147-151(2018).

    [17] Wang H D, Rivenson Y, Jin Y Y et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy[J]. Nature Methods, 16, 103-110(2019).

    [18] Nehme E, Weiss L E, Michaeli T et al. Deep-STORM: super-resolution single-molecule microscopy by deep learning[J]. Optica, 5, 458-464(2018).

    [19] Ouyang W, Aristov A, Lelek M et al. Deep learning massively accelerates super-resolution localization microscopy[J]. Nature Biotechnology, 36, 460-468(2018).

    [20] Rivenson Y, Göröcs Z, Günaydin H et al. Deep learning microscopy[J]. Optica, 4, 1437-1443(2017).

    [21] Heinrich L, Bogovic J A, Saalfeld S. Deep learning for isotropic super-resolution from non-isotropic 3D electron microscopy[M]. ∥Descoteaux M, Maier-Hein L, Franz A, et al. Medical image computing and computer-assisted intervention-MICCAI 2017. Lecture notes in computer science. Cham: Springer, 10434, 135-143(2017).

    [22] Wang H, Rivenson Y, Jin Y et al. Deep learning achieves super-resolution in fluorescence microscopy[J]. Biorxiv, 309641(2018).

    [23] Fang L Y, Cunefare D, Wang C et al. Automatic segmentation of nine retinal layer boundaries in OCT images of non-exudative AMD patients using deep learning and graph search[J]. Biomedical Optics Express, 8, 2732-2744(2017).

    [24] Lee C S, Baughman D M, Lee A Y. Deep learning is effective for classifying normal versus age-related macular degeneration OCT images[J]. Ophthalmology Retina, 1, 322-327(2017).

    [25] Schlegl T, Waldstein S M, Bogunovic H et al. Fully automated detection and quantification of macular fluid in OCT using deep learning[J]. Ophthalmology, 125, 549-558(2018).

    [26] Waller L, Tian L. Machine learning for 3D microscopy[J]. Nature, 523, 416-417(2015).

    [27] Nguyen T, Bui V, Nehmetallah G. Computational optical tomography using 3-D deep convolutional neural networks[J]. Optical Engineering, 57, 043111(2018).

    [28] Lü M, Wang H, Li G et al. Learning-based lensless imaging through optically thick scattering media[J]. Advanced Photonics, 1, 036002(2019). http://www.opticsjournal.net/Articles/Abstract?aid=OJcdeafc077f03aa8e

    [29] Li S, Deng M, Lee J et al. Imaging through glass diffusers using densely connected convolutional networks[J]. Optica, 5, 803-813(2018).

    [30] Horisaki R, Takagi R, Tanida J. Learning-based imaging through scattering media[J]. Optics Express, 24, 13738-13743(2016).

    [31] Satat G, Tancik M, Gupta O et al. Object classification through scattering media with deep learning on time resolved measurement[J]. Optics Express, 25, 17466-17479(2017).

    [32] Cheng S F, Li H H, Luo Y Q et al. Artificial intelligence-assisted light control and computational imaging through scattering media[J]. Journal of Innovative Optical Health Sciences, 12, 1930006(2019). http://www.opticsjournal.net/Articles/Abstract?aid=OJ190903000065C0FcIe

    [33] Goy A, Arthur K, Li S et al. Low photon count phase retrieval using deep learning[J]. Physical Review Letters, 121, 243902(2018).

    [34] Chen C, Chen Q F, Xu J et al. Learning to see in the dark. [C]∥2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 18-23, 2018, Salt Lake City, UT, USA. New York: IEEE, 3291-3300(2018).

    [35] Rivenson Y, Wang H D, Wei Z S et al. Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning[J]. Nature Biomedical Engineering, 3, 466-477(2019).

    [36] Rivenson Y, Liu T R, Wei Z S et al. PhaseStain: the digital staining of label-free quantitative phase microscopy images using deep learning[J]. Light: Science & Applications, 8, 23(2019).

    [37] Feng S J, Chen Q, Gu G H et al. Fringe pattern analysis using deep learning[J]. Advanced Photonics, 1, 025001(2019). http://www.opticsjournal.net/Articles/Abstract?aid=OJ1903140002050FcIeL

    [38] Yan K T, Yu Y J, Huang C T et al. Fringe pattern denoising based on deep learning[J]. Optics Communications, 437, 148-152(2019).

    [39] Wang K Q, Li Y, Qian K M et al. One-step robust deep learning phase unwrapping[J]. Optics Express, 27, 15100-15115(2019).

    [40] Feng S J, Zuo C, Yin W et al. Micro deep learning profilometry for high-speed 3D surface imaging[J]. Optics and Lasers in Engineering, 121, 416-427(2019).

    [41] Luo W, Schwing A G, Urtasun R. Efficient deep learning for stereo matching. [C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 26-July 1, 2016, Las Vegas, Nevada. New York: IEEE, 5695-5703(2016).

    [42] Kuznietsov Y, Stuckler J, Leibe B. Semi-supervised deep learning for monocular depth map prediction. [C]∥2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 21-26, 2017, Honolulu, HI, USA. New York: IEEE, 2215-2223(2017).

    [43] Gerchberg R W, Saxton W O. A practical algorithm for the determination of phase from image and diffraction plane pictures[J]. Optik, 35, 237-250(1972).

    [44] Gerchberg R W. Phase determination from image and diffraction plane pictures in the electron microscope[J]. Optik, 34, 275-284(1971).

    [45] Fienup J R, Wackerman C C. Phase-retrieval stagnation problems and solutions[J]. Journal of the Optical Society of America A, 3, 1897-1907(1986).

    [46] Seldin J H, Fienup J R. Numerical investigation of the uniqueness of phase retrieval[J]. Journal of the Optical Society of America A, 7, 412-427(1990).

    [47] Guizar-Sicairos M, Fienup J R. Understanding the twin-image problem in phase retrieval[J]. Journal of the Optical Society of America A, 29, 2367-2375(2012).

    [48] Wackerman C C, Yagle A E. Use of Fourier domain real-plane zeros to overcome a phase retrieval stagnation[J]. Journal of the Optical Society of America A, 8, 1898-1904(1991).

    [49] Lu G, Zhang Z. Yu F T S, et al. Pendulum iterative algorithm for phase retrieval from modulus data[J]. Optical Engineering, 33, 548-555(1994).

    [50] Takajo H, Takahashi T, Kawanami H et al. Numerical investigation of the iterative phase-retrieval stagnation problem: territories of convergence objects and holes in their boundaries[J]. Journal of the Optical Society of America A, 14, 3175-3187(1997).

    [51] Misell D L. A method for the solution of the phase problem in electron microscopy[J]. Journal of Physics D: Applied Physics, 6, L6-L9(1973).

    [52] Zheng G A, Horstmeyer R, Yang C. Wide-field, high-resolution Fourier ptychographic microscopy[J]. Nature Photonics, 7, 739-745(2013).

    [53] Horstmeyer R, Chen R Y, Ou X Z et al. Solving ptychography with a convex relaxation[J]. New Journal of Physics, 17, 053044(2015).

    [54] Yeh L H, Dong J, Zhong J S et al. Experimental robustness of Fourier ptychography phase retrieval algorithms[J]. Optics Express, 23, 33214-33240(2015).

    [55] Zuo C, Sun J S, Chen Q. Adaptive step-size strategy for noise-robust Fourier ptychographic microscopy[J]. Optics Express, 24, 20724-20744(2016).

    [56] Pittman T B, Shih Y H, Strekalov D V et al. Optical imaging by means of two-photon quantum entanglement[J]. Physical Review A, 52, R3429-R3432(1995).

    [57] Bennink R S, Bentley S J, Boyd R W. “Two-photon” coincidence imaging with a classical source[J]. Physical Review Letters, 89, 113601(2002).

    [58] Gatti A, Brambilla E, Bache M et al. Ghost imaging with thermal light: comparing entanglement and classical correlation[J]. Physical Review Letters, 93, 093602(2004).

    [59] Sen P, Chen B, Garg G et al. Dual photography. [C]∥ACM SIGGRAPH 2005 Papers on-SIGGRAPH'05, July 31-August 4, 2005, Los Angeles, California. New York: ACM, 745-755(2005).

    [60] Dharmpal T, Laska J N, Michael B et al. A new compressive imaging camera architecture using optical-domain compression[J]. Proceedings of SPIE, 6065, 606509(2006).

    [61] Duarte M F, Davenport M A, Takhar D et al. Single-pixel imaging via compressive sampling[J]. IEEE Signal Processing Magazine, 25, 83-91(2008).

    [62] Sun B, Edgar M P, Bowman R et al. 3D computational imaging with single-pixel detectors[J]. Science, 340, 844-847(2013).

    [63] Vellekoop I M, Mosk A P. Focusing coherent light through opaque strongly scattering media[J]. Optics Letters, 32, 2309-2311(2007).

    [64] Popoff S M, Lerosey G, Carminati R et al. Measuring the transmission matrix in optics: an approach to the study and control of light propagation in disordered media[J]. Physical Review Letters, 104, 100601(2010).

    [65] Leith E N, Upatnieks J. Holographic imagery through diffusing media[J]. Journal of the Optical Society of America, 56, 523(1966).

    [66] Yaqoob Z, Psaltis D, Feld M S et al. Optical phase conjugation for turbidity suppression in biological samples[J]. Nature Photonics, 2, 110-115(2008).

    [67] Bertolotti J, van Putten E G, Blum C et al. Non-invasive imaging through opaque scattering layers[J]. Nature, 491, 232-234(2012). http://www.ncbi.nlm.nih.gov/pubmed/23135468

    [68] Yang W Q, Li G W, Situ G H. Imaging through scattering media with the auxiliary of a known reference object[J]. Scientific Reports, 8, 9614(2018).

    [69] Lü M, Wang H, Li G et al. Learning-based lensless imaging through optically thick scattering media[J]. Advanced Photonics, 1, 036002(2019). http://www.opticsjournal.net/Articles/Abstract?aid=OJcdeafc077f03aa8e

    [70] Wolf E. Three-dimensional structure determination of semi-transparent objects from holographic data[J]. Optics Communications, 1, 153-156(1969).

    [71] Kak A C, Slaney M[M]. Principles of computerized tomographic imaging(2001).

    [72] Haeberlé O, Belkebir K, Giovaninni H et al. Tomographic diffractive microscopy: basics, techniques and perspectives[J]. Journal of Modern Optics, 57, 686-699(2010).

    [73] Rappaz B, Marquet P, Cuche E et al. Measurement of the integral refractive index and dynamic cell morphometry of living cells with digital holographic microscopy[J]. Optics Express, 13, 9361-9373(2005).

    [74] Lauer V. New approach to optical diffraction tomography yielding a vector equation of diffraction tomography and a novel tomographic microscope[J]. Journal of Microscopy, 205, 165-176(2002).

    [75] Choi W. Tomographic phase microscopy and its biological applications[J]. 3D Research, 3, 5(2012).

    [76] Charrière F, Marian A, Montfort F et al. Cell refractive index tomography by digital holographic microscopy[J]. Optics Letters, 31, 178-180(2006).

    [77] Charrière F, Pavillon N, Colomb T et al. Living specimen tomography by digital holographic microscopy: morphometry of testate amoeba[J]. Optics Express, 14, 7005-7013(2006).

    [78] Choi W, Fang-Yen C, Badizadegan K et al. Tomographic phase microscopy[J]. Nature Methods, 4, 717-719(2007).

    [79] Sung Y, Choi W, Fang-Yen C et al. Optical diffraction tomography for high resolution live cell imaging[J]. Optics Express, 17, 266-277(2009).

    [80] Kim K, Yoon H, Diez-Silva M et al. High-resolution three-dimensional imaging of red blood cells parasitized by Plasmodium falciparum and in situ hemozoin crystals using optical diffraction tomography[J]. Journal of Biomedical Optics, 19, 011005(2014).

    [81] Cotte Y, Toy F, Jourdain P et al. Marker-free phase nanoscopy[J]. Nature Photonics, 7, 113-117(2013).

    [82] Devaney A. A filtered backpropagation algorithm for diffraction tomography[J]. Ultrasonic Imaging, 4, 336-350(1982).

    [83] Barty A, Nugent K A, Roberts A et al. Quantitative phase tomography[J]. Optics Communications, 175, 329-336(2000).

    [84] Soto J M, Rodrigo J A, Alieva T. Label-free quantitative 3D tomographic imaging for partially coherent light microscopy[J]. Optics Express, 25, 15699-15712(2017).

    [85] Li J, Chen Q, Sun J et al. Three-dimensional tomographic microscopy technique with multi-frequency combination with partially coherent illuminations[J]. Biomedical Optics Express, 9, 2526-2542(2018).

    [86] Soto J M, Rodrigo J A, Alieva T. Optical diffraction tomography with fully and partially coherent illumination in high numerical aperture label-free microscopy [Invited][J]. Applied Optics, 57, A205-A214(2018).

    [87] Horstmeyer R, Chung J, Ou X et al. Diffraction tomography with Fourier ptychography[J]. Optica, 3, 827-835(2016).

    [88] Tian L, Waller L. 3D intensity and phase imaging from light field measurements in an LED array microscope[J]. Optica, 2, 104-111(2015).

    [89] Zuo C, Sun J, Li J et al. -05-26)[2019-11-20], org/abs/1904, 09386(2019). https://arxiv.

    [90] Kamilov U S, Papadopoulos I N, Shoreh M H et al. Learning approach to optical tomography[J]. Optica, 2, 517-522(2015).

    [92] Hell S W, Wichmann J. Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy[J]. Optics Letters, 19, 780-782(1994).

    [93] Betzig E, Patterson G H, Sougrat R et al. Imaging intracellular fluorescent proteins at nanometer resolution[J]. Science, 313, 1642-1645(2006).

    [94] Rust M J, Bates M, Zhuang X W. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM)[J]. Nature Methods, 3, 793-796(2006).

    [95] Zeiler M D, Fergus R. Visualizing and understanding convolutional networks[M]. ∥Fleet D, Pajdla T, Schiele B, et al. Computer vision-ECCV 2014. Lecture notes in computer science. Cham: Springer, 8689, 818-833(2014).

    [96] Shwartz-Ziv R. -04-29)[2019-11-20][J/OL]. Tishby N. Opening the black box of deep neural networks via information., org/abs/1703, 00810(2017). https://arxiv.

    [97] Goodfellow I, Pouget-Abadie J, Mirza M et al. Generative adversarial nets. [C]∥Advances in Neural Information Processing Systems, December 8-13, 2014, Montreal, Quebec, Canada. Canada: NIPS, 2672-2680(2014).

    [98] Xu Z B, Sun J. Model-driven deep-learning[J]. National Science Review, 5, 22-24(2018).

    [99] Lin H W, Tegmark M, Rolnick D. Why does deep and cheap learning work so well?[J]. Journal of Statistical Physics, 168, 1223-1247(2017).

    [101] Nguyen T Q, Weitekamp D, Anderson D et al. Topology classification with deep learning to improve real-time event selection at the LHC[J]. Computing and Software for Big Science, 3, 12(2019).

    [102] Perera P, Patel V M. Deep transfer learning for multiple class novelty detection. [C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 16-20, 2019, Long Beach, CA, USA. New York: IEEE, 11544-11552(2019).

    [103] Perera P, Nallapati R, Xiang B. OCGAN: one-class novelty detection using GANs with constrained latent representations. [C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 16-20, 2019, Long Beach, CA, USA. New York: IEEE, 2898-2906(2019).

    [104] Goodfellow I J, Shlens J, Szegedy C. Explaining, harnessing adversarial examples[J/OL]. -03-20)[2019-11-20], org/abs/1412, 6572(2015). https://arxiv.

    [105] DeVries P M R, Viégas F, Wattenberg M et al. Deep learning of aftershock patterns following large earthquakes[J]. Nature, 560, 632-634(2018).

    [106] Machine Learning[2019-11-20]. Deep learning of aftershock patterns following large earthquakes [2019-11-20].https://www.reddit.com/r/MachineLearning/comments/9bo9i9/r_deep_learning_of_aftershock_patterns_following/..

    CLP Journals

    [1] Qican Zhang, Zhoujie Wu. Three-dimensional imaging technique based on Gray-coded structured illumination[J]. Infrared and Laser Engineering, 2020, 49(3): 0303004

    Chao Zuo, Shijie Feng, Xiangyu Zhang, Jing Han, Chen Qian. Deep Learning Based Computational Imaging: Status, Challenges, and Future[J]. Acta Optica Sinica, 2020, 40(1): 0111003
    Download Citation