Researching | Advances in Deep Learning for Super-Resolution Microscopy（Invited）

Journals >Laser & Optoelectronics Progress >Volume 61 >Issue 16 >Page 1611002 > Article

Laser & Optoelectronics Progress
Vol. 61, Issue 16, 1611002 (2024)

Advances in Deep Learning for Super-Resolution Microscopy（Invited）

Xinyi Lu^1,2, Yu Huang³, Zitong Zhang⁴, Tianxiao Wu^1,2..., Hongjun Wu^1,2, Yongtao Liu^1,2,*, Zhong Fang^3,**, Chao Zuo^1,2,*** and Qian Chen^1,2|Show fewer author(s)

Author Affiliations

¹Smart Computational Imaging Laboratory, College of Electronic and Optical Engineering, Nanjing University of Science and Technology, Nanjing 210094, Jiangsu, China

²Key Laboratory of Spectral Imaging & Intelligent Sense, Nanjing University of Science and Technology, Nanjing 210094, Jiangsu, China

³School of Mechanical Engineering, Nanjing University of Science and Technology, Nanjing 210094, Jiangsu, China

⁴Infection Management Department of Shenzhen Sami Medical Center (Shenzhen Fourth People's Hospital), Shenzhen 518118, Guangdong, China

show less

DOI: 10.3788/LOP241455 Cite this Article Set citation alerts

Xinyi Lu, Yu Huang, Zitong Zhang, Tianxiao Wu, Hongjun Wu, Yongtao Liu, Zhong Fang, Chao Zuo, Qian Chen. Advances in Deep Learning for Super-Resolution Microscopy（Invited）[J]. Laser & Optoelectronics Progress, 2024, 61(16): 1611002 Copy Citation Text

show less

References

[1] Chen X D, Zou C L, Gong Z J et al. Subdiffraction optical manipulation of the charge state of nitrogen vacancy center in diamond[J]. Light: Science & Applications, 4, e230(2015).

[2] Yang T J, Luo Y R, Ji W et al. Advancing biological super-resolution microscopy through deep learning: a brief review[J]. Biophysics Reports, 7, 253-266(2021).

[3] Rust M J, Bates M, Zhuang X W. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM)[J]. Nature Methods, 3, 793-795(2006).

[4] Betzig E, Patterson G H, Sougrat R et al. Imaging intracellular fluorescent proteins at nanometer resolution[J]. Science, 313, 1642-1645(2006).

[5] Chen C H, Wang F, Wen S H et al. Multi-photon near-infrared emission saturation nanoscopy using upconversion nanoparticles[J]. Nature Communications, 9, 3290(2018).

[6] Denk W, Strickler J H, Webb W W. Two-photon laser scanning fluorescence microscopy[J]. Science, 248, 73-76(1990).

[7] Liu Y T, Wang F, Lu H X et al. Super-resolution mapping of single nanoparticles inside tumor spheroids[J]. Small, 16, e1905572(2020).

[8] Liu Y T, Wen S H, Wang F et al. Population control of upconversion energy transfer for stimulation emission depletion nanoscopy[J]. Advanced Science, 10, e2205990(2023).

[9] Cao Y Y, Li X P, Gu M. Super-resolution nanofabrication with metal-ion doped hybrid material through an optical dual-beam approach[J]. Applied Physics Letters, 105, 263102(2014).

[10] Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 60, 84-90(2017).

[11] Friedrich M, Gan Q, Ermolayev V et al. STED-SPIM: stimulated emission depletion improves sheet illumination microscopy resolution[J]. Biophysical Journal, 100, L43-L45(2011).

[12] Ronneberger O, Fischer P, Brox T, Navab N, Hornegger J, Wells W M et al. U-Net: convolutional networks for biomedical image segmentation[M]. Medical image computing and computer-assisted intervention- MICCAI 2015. Lecture notes in computer science, 9351, 234-241(2015).

[13] He K M, Zhang X Y, Ren S Q et al. Deep residual learning for image recognition[C], 770-778(2016).

[14] Ren S Q, He K M, Girshick R et al. Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39, 1137-1149(2017).

[15] Wang J L, Yan W, Zhang J et al. New advances in the research of stimulated emission depletion super-resolution microscopy[J]. Acta Physica Sinica, 69, 108702(2020).

[16] Schermelleh L, Ferrand A, Huser T et al. Super-resolution microscopy demystified[J]. Nature Cell Biology, 21, 72-84(2019).

[17] Ebrahimi V, Stephan T, Kim J et al. Deep learning enables fast, gentle STED microscopy[J]. Communications Biology, 6, 674(2023).

[18] Li M Z. Deep adversarial network for super stimulated emission depletion imaging[J]. Journal of Nanophotonics, 14, 016009(2020).

[19] An S, Dan D, Yu X H et al. Progress and prospect of research on single-molecule localization super-resolution microscopy (invited review)[J]. Acta Photonica Sinica, 49, 0918001(2020).

[20] Nehme E, Weiss L E, Michaeli T et al. Deep-STORM: super-resolution single-molecule microscopy by deep learning[J]. Optica, 5, 458-464(2018).

[21] Gaire S K, Zhang Y, Li H Y et al. Accelerating multicolor spectroscopic single-molecule localization microscopy using deep learning[J]. Biomedical Optics Express, 11, 2705-2721(2020).

[22] Möckl L, Roy A R, Petrov P N et al. Accurate and rapid background estimation in single-molecule localization microscopy using the deep neural network BGNet[J]. Proceedings of the National Academy of Sciences of the United States of America, 117, 60-67(2020).

[23] Kim T, Moon S, Xu K. Information-rich localization microscopy through machine learning[J]. Nature Communications, 10, 1996(2019).

[24] Jin L H, Liu B, Zhao F Q et al. Deep learning enables structured illumination microscopy with low light levels and enhanced speed[J]. Nature Communications, 11, 1934(2020).

[25] Shah Z H, Müller M, Wang T C et al. Deep-learning based denoising and reconstruction of super-resolution structured illumination microscopy images[J]. Photonics Research, 9, B168-B181(2021).

[26] Burns Z, Liu Z W. Untrained, physics-informed neural networks for structured illumination microscopy[J]. Optics Express, 31, 8714-8724(2023).

[27] Huang B Y, Li J, Yao B W et al. Enhancing image resolution of confocal fluorescence microscopy with deep learning[J]. PhotoniX, 4, 2(2023).

[28] Wang Y F, Kuang C F, Gu Z T et al. Time-gated stimulated emission depletion nanoscopy[J]. Optical Engineering, 52, 093107(2013).

[29] Hell S W, Wichmann J. Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy[J]. Optics Letters, 19, 780-782(1994).

[30] Heller I, Sitters G, Broekmans O D et al. STED nanoscopy combined with optical tweezers reveals protein dynamics on densely covered DNA[J]. Nature Methods, 10, 910-916(2013).

[31] Liu Y T, Zhou J J, Wen S H et al. On-chip mirror enhanced multiphoton upconversion super-resolution microscopy[J]. Nano Letters, 23, 5514-5519(2023).

[32] Takasaki K T, Ding J B, Sabatini B L. Live-cell superresolution imaging by pulsed STED two-photon excitation microscopy[J]. Biophysical Journal, 104, 770-777(2013).

[33] Kilian N, Goryaynov A, Lessard M D et al. Assessing photodamage in live-cell STED microscopy[J]. Nature Methods, 15, 755-756(2018).

[34] Tortarolo G, Castello M, Diaspro A et al. Evaluating image resolution in stimulated emission depletion microscopy[J]. Optica, 5, 32-35(2018).

[35] Chen Y I, Chang Y J, Sun Y S et al. Spatial resolution enhancement in photon-starved STED imaging using deep learning-based fluorescence lifetime analysis[J]. Nanoscale, 15, 9449-9456(2023).

[36] Pavani S R P, Thompson M A, Biteen J S et al. Three-dimensional, single-molecule fluorescence imaging beyond the diffraction limit by using a double-helix point spread function[J]. Proceedings of the National Academy of Sciences of the United States of America, 106, 2995-2999(2009).

[37] Lew M D, Lee S F, Badieirostami M et al. Corkscrew point spread function for far-field three-dimensional nanoscale localization of pointlike objects[J]. Optics Letters, 36, 202-204(2011).

[38] Ji Y H, Chen D N, Wu H Z et al. Localizing axial dense emitters based on single-helix point spread function and deep learning[EB/OL]. http://arxiv.org/abs/2402.06863v2

[39] Pan W H, Li W, Qu J H et al. Research progress on organic fluorescent probes for single molecule localization microscopy[J]. Chinese Journal of Applied Chemistry, 36, 269-281(2019).

[40] Hess S T, Girirajan T P K, Mason M D. Ultra-high resolution imaging by fluorescence photoactivation localization microscopy[J]. Biophysical Journal, 91, 4258-4272(2006).

[41] Colabrese S, Castello M, Vicidomini G et al. Machine learning approach for single molecule localisation microscopy[J]. Biomedical Optics Express, 9, 1680-1691(2018).

[42] Sahel Y B, Eldar Y. Self-STORM: deep unrolled self-supervised learning for super-resolution microscopy[EB/OL]. http://arxiv.org/abs/2403.16974v1

[43] Li J Y, Tong G, Pan Y N et al. Spatial and temporal super-resolution for fluorescence microscopy by a recurrent neural network[J]. Optics Express, 29, 15747-15763(2021).

[44] Boyd N, Jonas E, Babcock H et al. DeepLoco: fast 3D localization microscopy using neural networks[EB/OL]. https://www.biorxiv.org/content/10.1101/267096v1

[45] Zelger P, Kaser K, Rossboth B et al. Three-dimensional localization microscopy using deep learning[J]. Optics Express, 26, 33166-33179(2018).

[46] Speiser A, Müller L R, Hoess P et al. Deep learning enables fast and dense single-molecule localization with high accuracy[J]. Nature Methods, 18, 1082-1090(2021).

[47] Ouyang W, Aristov A, Lelek M et al. Deep learning massively accelerates super-resolution localization microscopy[J]. Nature Biotechnology, 36, 460-468(2018).

[48] Isola P, Zhu J Y, Zhou T H et al. Image-to-image translation with conditional adversarial networks[C], 5967-5976(2017).

[49] Saguy A, Alalouf O, Opatovski N et al. DBlink: dynamic localization microscopy in super spatiotemporal resolution via deep learning[J]. Nature Methods, 20, 1939-1948(2023).

[50] Cascarano P, Comes M C, Sebastiani A et al. DeepCEL0 for 2D single-molecule localization in fluorescence microscopy[J]. Bioinformatics, 38, 1411-1419(2022).

[51] Nehme E, Freedman D, Gordon R et al. DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning[J]. Nature Methods, 17, 734-740(2020).

[52] Zhang P Y, Liu S, Chaurasia A et al. Analyzing complex single-molecule emission patterns with deep learning[J]. Nature Methods, 15, 913-916(2018).

[53] Hershko E, Weiss L E, Michaeli T et al. Multicolor localization microscopy and point-spread-function engineering by deep learning[J]. Optics Express, 27, 6158-6183(2019).

[54] Jeong D, Kim D. Super-resolution fluorescence microscopy-based single-molecule spectroscopy[J]. Bulletin of the Korean Chemical Society, 43, 316-327(2022).

[55] Kim G H, Chung J, Park H et al. Single-molecule sensing by grating-based spectrally resolved super-resolution microscopy[J]. Bulletin of the Korean Chemical Society, 42, 270-278(2021).

[56] Zhang Z Y, Kenny S J, Hauser M et al. Ultrahigh-throughput single-molecule spectroscopy and spectrally resolved super-resolution microscopy[J]. Nature Methods, 12, 935-938(2015).

[57] Zhang Z Y, Zhang Y, Ying L et al. Machine-learning based spectral classification for spectroscopic single-molecule localization microscopy[J]. Optics Letters, 44, 5864-5867(2019).

[58] Brunstein M, Wicker K, Hérault K et al. Full-field dual-color 100-nm super-resolution imaging reveals organization and dynamics of mitochondrial and ER networks[J]. Optics Express, 21, 26162-26173(2013).

[59] Schermelleh L, Carlton P M, Haase S et al. Subdiffraction multicolor imaging of the nuclear periphery with 3D structured illumination microscopy[J]. Science, 320, 1332-1336(2008).

[60] Turcotte R, Liang Y J, Tanimoto M et al. Dynamic super-resolution structured illumination imaging in the living brain[J]. Proceedings of the National Academy of Sciences of the United States of America, 116, 9586-9591(2019).

[61] Ma Y, Wen K, Liu M et al. Recent advances in structured illumination microscopy[J]. Journal of Physics: Photonics, 3, 024009(2021).

[62] Saxena M, Eluru G, Gorthi S S. Structured illumination microscopy[J]. Advances in Optics and Photonics, 7, 241-275(2015).

[63] Ling C, Zhang C L, Wang M Q et al. Fast structured illumination microscopy via deep learning[J]. Photonics Research, 8, 1350-1359(2020).

[64] Qiao C, Chen X Y, Zhang S W et al. 3D structured illumination microscopy via channel attention generative adversarial network[J]. IEEE Journal of Selected Topics in Quantum Electronics, 27, 6801711(2021).

[65] Qiao C, Li D, Guo Y T et al. Evaluation and development of deep neural networks for image super-resolution in optical microscopy[J]. Nature Methods, 18, 194-202(2021).

[66] Cheng X, Li J, Dai Q et al. Fast and lightweight network for single frame structured illumination microscopy super-resolution[J]. IEEE Transactions on Instrumentation and Measurement, 71, 5007711(2022).

[67] Müller M, Mönkemöller V, Hennig S et al. Open-source image reconstruction of super-resolution structured illumination microscopy data in ImageJ[J]. Nature Communications, 7, 10980(2016).

[68] Wicker K, Mandula O, Best G et al. Phase optimisation for structured illumination microscopy[J]. Optics Express, 21, 2032-2049(2013).

[69] Delp S L, Anderson F C, Arnold A S et al. OpenSim: open-source software to create and analyze dynamic simulations of movement[J]. IEEE Transactions on Bio-Medical Engineering, 54, 1940-1950(2007).

[70] Chen Y B, Liu Q Q, Zhang J F et al. Deep learning enables contrast-robust super-resolution reconstruction in structured illumination microscopy[J]. Optics Express, 32, 3316-3328(2024).

[71] Cao R Z, Chen Y H, Liu W J et al. Inverse matrix based phase estimation algorithm for structured illumination microscopy[J]. Biomedical Optics Express, 9, 5037-5051(2018).

[72] Liao J H, Zhang C S, Xu X C et al. Deep-MSIM: fast image reconstruction with deep learning in multifocal structured illumination microscopy[J]. Advanced Science, 10, e2300947(2023).

[73] Gustafsson M G L, Shao L, Carlton P M et al. Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination[J]. Biophysical Journal, 94, 4957-4970(2008).

[74] Wang H D, Rivenson Y, Jin Y Y et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy[J]. Nature Methods, 16, 103-110(2019).

Get PDF(in Chinese)

Figures&Tables (13)

References (74)

Copy Citation Text

Xinyi Lu, Yu Huang, Zitong Zhang, Tianxiao Wu, Hongjun Wu, Yongtao Liu, Zhong Fang, Chao Zuo, Qian Chen. Advances in Deep Learning for Super-Resolution Microscopy（Invited）[J]. Laser & Optoelectronics Progress, 2024, 61(16): 1611002

Download Citation

Set citation alerts for the article

Set citation alerts for the article

Save the article for my favorites

Paper Information

Recommended Topics

laser devices and laser physics

Lasers and Laser Optics

laser manufacturing

Instrumentation, Measurement and Metrology

Set citation alerts for the article

Please enter your email address