Fast structured illumination microscopy via deep learning

Chang Ling; Chonglei Zhang; Mingqun Wang; Fanfei Meng; Luping Du; Xiaocong Yuan

doi:10.1364/PRJ.396122

[1] E. Abbe. Contributions to the theory of the microscope and that microscopic perception. Arch. Microsc. Anat., 9, 413-468(1873).

[2] M. J. Rust, M. Bates, X. Zhuang. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM). Nat. Methods, 3, 793-795(2006).

[3] M. Bates, B. Huang, G. T. Dempsey, X. Zhuang. Multicolor super-resolution imaging with photo-switchable fluorescent probes. Science, 317, 1749-1753(2007).

[4] S. T. Hess, T. P. K. Girirajan, M. D. Mason. Ultra-high resolution imaging by fluorescence photoactivation localization microscopy. Biophys. J., 91, 4258-4272(2006).

[5] H. Shroff, C. G. Galbraith, J. A. Galbraith, E. Betzig. Live-cell photoactivated localization microscopy of nanoscale adhesion dynamics. Nat. Methods, 5, 417-423(2008).

[6] M. G. L. Gustafsson, D. A. Agard, J. W. Sedat. Doubling the lateral resolution of wide-field fluorescence microscopy using structured illumination. Proc. SPIE, 3919, 141-150(2000).

[7] M. G. L. Gustafsson, L. Shao, P. M. Carlton, C. J. R. Wang, I. N. Golubovskaya, W. Z. Cande, D. A. Agard, J. W. Sedat. Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination. Biophys. J., 94, 4957-4970(2008).

[8] T. A. Klar, S. Jakobs, M. Dyba, A. Egner, S. W. Hell. Fluorescence microscopy with diffraction resolution barrier broken by stimulated emission. Proc. Natl. Acad. Sci. USA, 97, 8206-8210(2000).

[9] E. Betzig, G. H. Patterson, R. Sougrat, O. W. Lindwasser, S. Olenych, J. S. Bonifacino, M. W. Davidson, J. Lippincott-Schwartz, H. F. Hess. Imaging intracellular fluorescent proteins at nanometer resolution. Science, 313, 1642-1645(2006).

[10] H. Linnenbank, T. Steinle, F. Morz, M. Floss, C. Han, A. Glidle, H. Giessen. Robust and rapidly tunable light source for SRS/CARS microscopy with low-intensity noise. Adv. Photon., 1, 055001(2019).

[11] P. Fei, J. Nie, J. Lee, Y. Ding, S. Li, H. Zhang, M. Hagiwara, T. Yu, T. Segura, C.-M. Ho, D. Zhu, T. K. Hsiai. Subvoxel light-sheet microscopy for high-resolution high-throughput volumetric imaging of large biomedical specimens. Adv. Photon., 1, 016002(2019).

[12] E. Narimanov. Resolution limit of label-free far-field microscopy. Adv. Photon., 1, 056003(2019).

[13] E. F. Fornasiero, K. Wicker, S. O. Rizzoli. Super-resolution fluorescence microscopy using structured illumination. Super-Resolution Microscopy Techniques in the Neurosciences, 133-165(2014).

[14] M. G. L. Gustafsson. Nonlinear structured-illumination microscopy: wide-field fluorescence imaging with theoretically unlimited resolution. Proc. Natl. Acad. Sci. USA, 102, 13081-13086(2005).

[15] F. Orieux, E. Sepulveda, V. Loriette, B. Dubertret, J.-C. Olivo-Marin. Bayesian estimation for optimized structured illumination microscopy. IEEE Trans. Image Process., 21, 601-614(2012).

[16] S. Dong, J. Liao, K. Guo, L. Bian, J. Suo, G. Zheng. Resolution doubling with a reduced number of image acquisitions. Biomed. Opt. Express, 6, 2946-2952(2015).

[17] A. Lal, C. Shan, K. Zhao, W. Liu, X. Huang, W. Zong, L. Chen, P. Xi. A frequency domain SIM reconstruction algorithm using reduced number of images. IEEE Trans. Image Process., 27, 4555-4570(2018).

[18] F. Strohl, C. F. Kaminski. Speed limits of structured illumination microscopy. Opt. Lett., 42, 2511-2514(2017).

[19] W. H. Richardson. Bayesian-based iterative method of image restoration. J. Opt. Soc. Am., 62, 55-59(1972).

[20] M. Ingaramo, A. G. York, E. Hoogendoorn, M. Postma, H. Shroff, G. H. Patterson. Richardson-Lucy deconvolution as a general tool for combining images with complementary strengths. Chem. Phys. Chem., 15, 794-800(2014).

[21] M. I. Jordan, T. M. Mitchell. Machine learning: trends, perspectives, and prospects. Science, 349, 255-260(2015).

[22] Y. LeCun, Y. Bengio, G. Hinton. Deep learning. Nature, 521, 436-444(2015).

[23] A. Krizhevsky, I. Sutskever, G. E. Hinton. ImageNet classification with deep convolutional neural networks. Commun. ACM, 60, 84-90(2017).

[24] Y. Rivenson, Z. Gorocs, H. Gunaydin, Y. Zhang, H. Wang, A. Ozcan. Deep learning microscopy. Optica, 4, 1437-1443(2017).

[25] W. Ouyang, A. Aristov, M. Lelek, X. Hao, C. Zimmer. Deep learning massively accelerates super-resolution localization microscopy. Nat. Biotechnol., 36, 460-468(2018).

[26] E. Nehme, L. E. Weiss, T. Michaeli, Y. Shechtman. Deep-STORM: super-resolution single-molecule microscopy by deep learning. Optica, 5, 458-464(2018).

[27] N. Thanh, Y. Xue, Y. Li, L. Tian, G. Nehmetallah. Deep learning approach to Fourier ptychographic microscopy. Opt. Express, 26, 26470-26484(2018).

[28] Z. Ghahramani, I. J. Goodfellow, M. Welling, J. Pouget-Abadie, M. Mirza, C. Cortes, N. D. Lawrence, B. Xu, K. Q. Weinberger, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio. Generative adversarial nets. Proceedings of the 27th International Conference on Neural Information Processing Systems, 2672-2680(2014).

[29] , J.-Y. Zhu, T. Park, P. Isola, A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. IEEE International Conference on Computer Vision, 2242-2251(2017).

[30] M. Mirza, S. Osindero. Conditional generative adversarial nets(2014).

[31] L. A. Gatys, A. S. Ecker, M. Bethge. Image style transfer using convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2414-2423(2016).

[32] , P. Isola, J.-Y. Zhu, T. Zhou, A. A. Efros. Image-to-image translation with conditional adversarial networks. 30th IEEE Conference on Computer Vision and Pattern Recognition, 5967-5976(2017).

[33] B. Leibe, C. Li, J. Matas, M. Wand, N. Sebe, M. Welling. Precomputed real-time texture synthesis with Markovian generative adversarial networks. Computer Vision—European Conference on Computer Vision (ECCV), 702-716(2016).

[34] K. Daniilidis, N. Sundaram, P. Maragos, T. Brox, N. Paragios, K. Keutzer. Dense point trajectories by GPU-accelerated large displacement optical flow. Computer Vision—European Conference on Computer Vision (ECCV), 438-451(2010).

[35] C. Godard, , O. Mac Aodha, G. J. Brostow. Unsupervised monocular depth estimation with left-right consistency. 30th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 6602-6611(2017).

[36] , K. He, X. Zhang, S. Ren, J. Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770-778(2016).

[37] A. Lal, C. Shan, P. Xi. Structured illumination microscopy image reconstruction algorithm. IEEE J. Sel. Top. Quantum Electron., 22, 6803414(2016).

[38] M. Mueller, V. Moenkemoeller, S. Hennig, W. Huebner, T. Huser. Open-source image reconstruction of super-resolution structured illumination microscopy data in ImageJ. Nat. Commun., 7, 10980(2016).

[39] Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process., 13, 600-612(2004).

[40] M. B. Matthews, Z. Wang, E. P. Simoncelli, A. C. Bovik. Multi-scale structural similarity for image quality assessment. Conference Record of the 37th Asilomar Conference on Signals, Systems & Computers, 1398-1402(2003).

[41] H. Wang, Y. Rivenson, Y. Jin, Z. Wei, R. Gao, H. Gunaydin, L. A. Bentolila, C. Kural, A. Ozcan. Deep learning enables cross-modality super-resolution in fluorescence microscopy. Nat. Methods, 16, 103-110(2019).

[42] L. Jin, B. Liu, F. Zhao, S. Hahn, B. Dong, R. Song, T. C. Elston, Y. Xu, K. M. Hahn. Deep learning enables structured illumination microscopy with low light levels and enhanced speed. Nat. Commun., 11, 1934(2020).

CLP Journals