• Laser & Optoelectronics Progress
  • Vol. 58, Issue 24, 2400007 (2021)
Qingshuang Lu1, Luhong Jin2、*, and Yingke Xu2
Author Affiliations
  • 1Department of Humanities and Tourism, Zhejiang Institute of Economics and Trade, Hangzhou , Zhejiang 310018, China
  • 2Key Laboratory of Biomedical Engineering, Ministry of Education, Zhejiang Province Key Laboratory of Cardio-Cerebral Vascular Detection Technology and Medicinal Effectiveness Appraisal, Department of Biomedical Engineering, Zhejiang University, Hangzhou , Zhejiang 310027, China
  • show less
    DOI: 10.3788/LOP202158.2400007 Cite this Article Set citation alerts
    Qingshuang Lu, Luhong Jin, Yingke Xu. Progress on Applications of Deep Learning in Super-Resolution Microscopy Imaging[J]. Laser & Optoelectronics Progress, 2021, 58(24): 2400007 Copy Citation Text show less
    References

    [1] Xu Y K, Toomre D K, Bogan J S et al. Excess cholesterol inhibits glucose-stimulated fusion pore dynamics in insulin exocytosis[J]. Journal of Cellular and Molecular Medicine, 21, 2950-2962(2017).

    [2] Xiao X, Geyer V F, Bowne-Anderson H et al. Automatic optimal filament segmentation with sub-pixel accuracy using generalized linear models and B-spline level-sets[J]. Medical Image Analysis, 32, 157-172(2016).

    [3] Valm A M, Cohen S, Legant W R et al. Applying systems-level spectral imaging and analysis to reveal the organelle interactome[J]. Nature, 546, 162-167(2017).

    [4] Rowland A A, Chitwood P J, Phillips M J et al. ER contact sites define the position and timing of endosome fission[J]. Cell, 159, 1027-1041(2014).

    [5] Lewis S C, Uchiyama L F, Nunnari J. ER-mitochondria contacts couple mtDNA synthesis with mitochondrial division in human cells[J]. Science, 353, aaf5549(2016).

    [6] Pan W H, Li W, Qu J H et al. Research progress on organic fluorescent probes for single molecule localization microscopy[J]. Chinese Journal of Applied Chemistry, 36, 269-281(2019).

    [7] Vangindertael J, Camacho R, Sempels W et al. An introduction to optical super-resolution microscopy for the adventurous biologist[J]. Methods and Applications in Fluorescence, 6, 022003(2018).

    [8] Shao L, Kner P, Rego E H et al. Super-resolution 3D microscopy of live whole cells using structured illumination[J]. Nature Methods, 8, 1044-1046(2011).

    [9] Wildanger D, Medda R, Kastrup L et al. A compact STED microscope providing 3D nanoscale resolution[J]. Journal of Microscopy, 236, 35-43(2009).

    [10] Markwirth A, Lachetta M, Mönkemöller V et al. Video-rate multi-color structured illumination microscopy with simultaneous real-time reconstruction[J]. Nature Communications, 10, 4315(2019).

    [11] Belthangady C, Royer L A. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction[J]. Nature Methods, 16, 1215-1225(2019).

    [12] Wang F, Wang H, Bian Y M et al. Applications of deep learning in computational imaging[J]. Acta Optica Sinica, 40, 0111002(2020).

    [13] Zuo C, Feng S J, Zhang X Y et al. Deep learning based computational imaging: status, challenges, and future[J]. Acta Optica Sinica, 40, 0111003(2020).

    [14] Mishin A S, Lukyanov K A. Live-cell super-resolution fluorescence microscopy[J]. Biochemistry (Moscow), 84, 19-31(2019).

    [15] Huang X, Fan J, Li L et al. Fast, long-term, super-resolution imaging with Hessian structured illumination microscopy[J]. Nature Biotechnology, 36, 451-459(2018).

    [16] Dong D, Huang X, Li L et al. Super-resolution fluorescence-assisted diffraction computational tomography reveals the three-dimensional landscape of the cellular organelle interactome[J]. Light, Science & Applications, 9, 11(2020).

    [17] Chen Y, Liu W, Zhang Z et al. Multi-color live-cell super-resolution volume imaging with multi-angle interference microscopy[J]. Nature Communications, 9, 4818(2018).

    [18] Karras C, Smedh M, Förster R et al. Successful optimization of reconstruction parameters in structured illumination microscopy: a practical guide[J]. Optics Communications, 436, 69-75(2019).

    [19] Perez V, Chang B J, Stelzer E H. Optimal 2D-SIM reconstruction by two filtering steps with Richardson-Lucy deconvolution[J]. Scientific Reports, 6, 37149(2016).

    [20] Lal A, Shan C Y, Zhao K et al. A frequency domain SIM reconstruction algorithm using reduced number of images[J]. IEEE Transactions on Image Processing, 27, 4555-4570(2018).

    [21] Ball G, Demmerle J, Kaufmann R et al. SIMcheck: a toolbox for successful super-resolution structured illumination microscopy[J]. Scientific Reports, 5, 15915(2015).

    [22] Li Y Z, Li C K, Hao X et al. Review and prospect for single molecule localization microscopy[J]. Laser & Optoelectronics Progress, 57, 240002(2020).

    [23] Biteen J, Willets K A. Introduction: super-resolution and single-molecule imaging[J]. Chemical Reviews, 117, 7241-7243(2017).

    [24] Rust M J, Bates M, Zhuang X. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM)[J]. Nature Methods, 3, 793-795(2006).

    [25] Betzig E, Patterson G H, Sougrat R et al. Imaging intracellular fluorescent proteins at nanometer resolution[J]. Science, 313, 1642-1645(2006).

    [26] Hess S T, Girirajan T P K, Mason M D. Ultra-high resolution imaging by fluorescence photoactivation localization microscopy[J]. Biophysical Journal, 91, 4258-4272(2006).

    [27] Lampe A, Haucke V, Sigrist S J et al. Multi-colour direct STORM with red emitting carbocyanines[J]. Biology of the Cell, 104, 229-237(2012).

    [28] Wolter S, Schüttpelz M, Tscherepanow M et al. Real-time computation of subdiffraction-resolution fluorescence images[J]. Journal of Microscopy, 237, 12-22(2010).

    [29] Köthe U, Herrmannsdörfer F, Kats I et al. SimpleSTORM: a fast, self-calibrating reconstruction algorithm for localization microscopy[J]. Histochemistry and Cell Biology, 141, 613-627(2014).

    [30] Holden S J, Uphoff S, Kapanidis A N. DAOSTORM: an algorithm for high- density super-resolution microscopy[J]. Nature Methods, 8, 279-280(2011).

    [31] Henriques R, Lelek M, Fornasiero E F et al. QuickPALM: 3D real-time photoactivation nanoscopy image processing in ImageJ[J]. Nature Methods, 7, 339-340(2010).

    [32] Khan A O, Simms V A, Pike J A et al. CRISPR-Cas9 mediated labelling allows for single molecule imaging and resolution[J]. Scientific Reports, 7, 8450(2017).

    [33] Gu L, Li Y, Zhang S et al. Molecular resolution imaging by repetitive optical selective exposure[J]. Nature Methods, 16, 1114-1118(2019).

    [34] Cnossen J, Hinsdale T, Thorsen R Ø et al. Localization microscopy at doubled precision with patterned illumination[J]. Nature Methods, 17, 59-63(2020).

    [35] Qu X, Wu D, Mets L et al. Nanometer-localized multiple single-molecule fluorescence microscopy[J]. Proceedings of the National Academy of Sciences of the United States of America, 101, 11298-11303(2004).

    [36] Wang Y F, Kuang C F, Cai H Q et al. Sub-diffraction imaging with confocal fluorescence microscopy by stochastic photobleaching[J]. Optics Communications, 312, 62-67(2014).

    [37] Cox S, Rosten E, Monypenny J et al. Bayesian localization microscopy reveals nanoscale podosome dynamics[J]. Nature Methods, 9, 195-200(2011).

    [38] Gustafsson N, Culley S, Ashdown G et al. Fast live-cell conventional fluorophore nanoscopy with ImageJ through super-resolution radial fluctuations[J]. Nature Communications, 7, 12471(2016).

    [39] Dertinger T, Colyer R, Iyer G et al. Fast, background-free, 3D super-resolution optical fluctuation imaging (SOFI)[J]. Proceedings of the National Academy of Sciences of the United States of America, 106, 22287-22292(2009).

    [40] Culley S, Tosheva K L, Matos Pereira P et al. SRRF: universal live-cell super-resolution microscopy[J]. The International Journal of Biochemistry & Cell Biology, 101, 74-79(2018).

    [41] Han Y B, Lu X, Zhang Z M et al. Ultra-fast, universal super-resolution radial fluctuations (SRRF) algorithm for live-cell super-resolution microscopy[J]. Optics Express, 27, 38337-38348(2019).

    [42] Liu Z, Lavis L D, Betzig E. Imaging live-cell dynamics and structure at the single-molecule level[J]. Molecular Cell, 58, 644-659(2015).

    [43] Liu Z, Jin L, Chen J et al. A survey on applications of deep learning in microscopy image analysis[J]. Computers in Biology and Medicine, 134, 104523(2021).

    [44] Moen E, Bannon D, Kudo T et al. Deep learning for cellular image analysis[J]. Nature Methods, 16, 1233-1246(2019).

    [45] Durand A, Wiesner T, Gardner M A et al. A machine learning approach for online automated optimization of super-resolution optical microscopy[J]. Nature Communications, 9, 5247(2018).

    [46] Arigovindan M, Fung J C, Elnatan D et al. High-resolution restoration of 3D structures from widefield images with extreme low signal-to-noise-ratio[J]. Proceedings of the National Academy of Sciences of the United States of America, 110, 17344-17349(2013).

    [47] Liu J, Huang X, Chen L et al. Deep-learning-enhanced fluorescence microscopy via degeneration decoupling[J]. Optics Express, 28, 14859-14873(2020).

    [48] Ronneberger O, Fischer P, Brox T. U-net: convolutional networks for biomedical image segmentation[M]. Medical image computing and computer-assisted intervention-MICCAI 2015, 9351, 234-241(2015).

    [49] Wang H, Rivenson Y, Jin Y et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy[J]. Nature Methods, 16, 103-110(2019).

    [50] Buades A, Coll B, Morel J M. A non-local algorithm for image denoising[C], 60-65(2005).

    [51] Sage D, Donati L, Soulez F et al. DeconvolutionLab2: an open-source software for deconvolution microscopy[J]. Methods, 115, 28-41(2017).

    [52] Preibisch S, Amat F, Stamataki E et al. Efficient Bayesian-based multiview deconvolution[J]. Nature Methods, 11, 645-648(2014).

    [53] Weigert M, Schmidt U, Boothe T et al. Content-aware image restoration: pushing the limits of fluorescence microscopy[J]. Nature Methods, 15, 1090-1097(2018).

    [54] Jin L, Liu B, Zhao F et al. Deep learning enables structured illumination microscopy with low light levels and enhanced speed[J]. Nature Communications, 11, 1934(2020).

    [55] Gazagnes S, Soubies E, Blanc-Féraud L. High density molecule localization for super-resolution microscopy using CEL0 based sparse approximation[C], 28-31(2017).

    [56] Nehme E, Weiss L E, Michaeli T et al. Deep-STORM: super-resolution single-molecule microscopy by deep learning[J]. Optica, 5, 458-464(2018).

    [57] Rueden C T, Schindelin J, Hiner M C et al. ImageJ2: ImageJ for the next generation of scientific image data[J]. BMC Bioinformatics, 18, 529(2017).

    [58] Schindelin J, Arganda-Carreras I, Frise E et al. Fiji: an open-source platform for biological-image analysis[J]. Nature Methods, 9, 676-682(2012).

    [59] Ovesný M, Křížek P, Borkovec J et al. ThunderSTORM: a comprehensive ImageJ plug-in for PALM and STORM data analysis and super-resolution imaging[J]. Bioinformatics, 30, 2389-2390(2014).

    [60] Zelger P, Kaser K, Rossboth B et al. Three-dimensional localization microscopy using deep learning[J]. Optics Express, 26, 33166-33179(2018).

    [61] Wu Y, Rivenson Y, Wang H et al. Three-dimensional virtual refocusing of fluorescence microscopy images using deep learning[J]. Nature Methods, 16, 1323-1331(2019).

    [62] Kim T, Moon S, Xu K. Information-rich localization microscopy through machine learning[J]. Nature Communications, 10, 1996(2019).

    [63] Goodfellow I, Pouget-Abadie J, Mirza M. Generative adversarial nets[J]. Advances in Neural Information Processing Systems, 2672-2680(2014).

    [64] Ouyang W, Aristov A, Lelek M et al. Deep learning massively accelerates super-resolution localization microscopy[J]. Nature Biotechnology, 36, 460-468(2018).

    [65] Han L, Yin Z Z. Learning to transfer microscopy image modalities[J]. Machine Vision and Applications, 29, 1257-1267(2018).

    [66] Qiao C, Li D, Guo Y et al. Evaluation and development of deep neural networks for image super-resolution in optical microscopy[J]. Nature Methods, 18, 194-202(2021).

    [67] Nguyen J P, Shipley F B, Linder A N et al. Whole-brain calcium imaging with cellular resolution in freely behaving Caenorhabditis elegans[J]. Proceedings of the National Academy of Sciences of the United States of America, 113, E1074-E1081(2016).

    [68] Schrödel T, Prevedel R, Aumayr K et al. Brain-wide 3D imaging of neuronal activity in Caenorhabditis elegans with sculpted light[J]. Nature Methods, 10, 1013-1020(2013).

    [69] Tomer R, Lovett-Barron M, Kauvar I et al. SPED light sheet microscopy: fast mapping of biological system structure and function[J]. Cell, 163, 1796-1806(2015).

    [70] Fu Y, Winter P W, Rojas R et al. Axial superresolution via multiangle TIRF microscopy with sequential imaging and photobleaching[J]. Proceedings of the National Academy of Sciences of the United States of America, 113, 4368-4373(2016).

    [71] Jin L H, Zhou X X, Xiu P et al. Imaging and reconstruction of cell cortex structures near the cell surface[J]. Optics Communications, 402, 699-705(2017).

    [72] Jin L H, Wu J, Xiu P et al. High-resolution 3D reconstruction of microtubule structures by quantitative multi-angle total internal reflection fluorescence microscopy[J]. Optics Communications, 395, 16-23(2017).

    [73] Lim J, Ayoub A B, Psaltis D. Three-dimensional tomography of red blood cells using deep learning[J]. Advanced Photonics, 2, 026001(2020).

    [74] Franke C, Sauer M, van de Linde S. Photometry unlocks 3D information from 2D localization microscopy data[J]. Nature Methods, 14, 41-44(2017).

    [75] Li Y, Mund M, Hoess P et al. Real-time 3D single-molecule localization using experimental point spread functions[J]. Nature Methods, 15, 367-369(2018).

    [76] Sage D, Pham T A, Babcock H et al. Super-resolution fight club: assessment of 2D and 3D single-molecule localization microscopy software[J]. Nature Methods, 16, 387-395(2019).

    [77] Christiansen E M, Yang S J, Ando D M et al. In silico labeling: predicting fluorescent labels in unlabeled images[J]. Cell, 173, 792-803.e19(2018).

    [78] Nketia T A, Sailem H, Rohde G et al. Analysis of live cell images: methods, tools and opportunities[J]. Methods, 115, 65-79(2017).

    [79] Godec P, Pančur M, Ilenič N et al. Democratized image analytics by visual programming through integration of deep models and small-scale machine learning[J]. Nature Communications, 10, 4551(2019).

    [80] Sun J, Tárnok A, Su X. Deep learning-based single-cell optical image studies[J]. Cytometry. Part A, 97, 226-240(2020).

    [81] Mahmood F, Borders D, Chen R J et al. Deep adversarial training for multi-organ nuclei segmentation in histopathology images[J]. IEEE Transactions on Medical Imaging, 39, 3257-3267(2020).

    [82] Pan S J, Yang Q. A survey on transfer learning[J]. IEEE Transactions on Knowledge and Data Engineering, 22, 1345-1359(2010).

    [83] Midtvedt B, Helgadottir S, Argun A et al. Quantitative digital microscopy with deep learning[J]. Applied Physics Reviews, 8, 011310(2021).

    Qingshuang Lu, Luhong Jin, Yingke Xu. Progress on Applications of Deep Learning in Super-Resolution Microscopy Imaging[J]. Laser & Optoelectronics Progress, 2021, 58(24): 2400007
    Download Citation