Deep-learning based denoising and reconstruction of super-resolution structured illumination microscopy images

Zafran Hussain Shah; Marcel Müller; Tung-Cheng Wang; Philip Maurice Scheidig; Axel Schneider; Mark Schüttpelz; Thomas Huser; Wolfram Schenck

doi:10.1364/PRJ.416437

[1] L. Schermelleh, A. Ferrand, T. Huser, C. Eggeling, M. Sauer, O. Biehlmaier, G. P. Drummen. Super-resolution microscopy demystified. Nat. Cell Biol., 21, 72-84(2019).

[2] J. Demmerle, C. Innocent, A. J. North, G. Ball, M. Müller, E. Miron, A. Matsuda, I. M. Dobbie, Y. Markaki, L. Schermelleh. Strategic and practical guidelines for successful structured illumination microscopy. Nat. Protoc., 12, 988-1010(2017).

[3] R. Heintzmann, T. Huser. Super-resolution structured illumination microscopy. Chem. Rev., 117, 13890-13908(2017).

[4] M. G. Gustafsson. Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy. J. Microsc., 198, 82-87(2000).

[5] L. M. Hirvonen, K. Wicker, O. Mandula, R. Heintzmann. Structured illumination microscopy of a living cell. Eur. Biophys. J., 38, 807-812(2009).

[6] P. Kner, B. B. Chhun, E. R. Griffis, L. Winoto, M. G. Gustafsson. Super-resolution video microscopy of live cells by structured illumination. Nat. Methods, 6, 339-342(2009).

[7] L. Shao, P. Kner, E. H. Rego, M. G. Gustafsson. Super-resolution 3D microscopy of live whole cells using structured illumination. Nat. Methods, 8, 1044-1046(2011).

[8] L. Gao, L. Shao, C. D. Higgins, J. S. Poulton, M. Peifer, M. W. Davidson, X. Wu, B. Goldstein, E. Betzig. Noninvasive imaging beyond the diffraction limit of 3D dynamics in thickly fluorescent specimens. Cell, 151, 1370-1385(2012).

[9] R. Fiolka, L. Shao, E. H. Rego, M. W. Davidson, M. G. Gustafsson. Time-lapse two-color 3D imaging of live cells with doubled resolution using structured illumination. Proc. Natl. Acad. Sci. USA, 109, 5311-5315(2012).

[10] X. Huang, J. Fan, L. Li, H. Liu, R. Wu, Y. Wu, L. Wei, H. Mao, A. Lal, P. Xi, L. Tang, Y. Zhang, Y. Liu, S. Tan, L. Chen. Fast, long-term, super-resolution imaging with Hessian structured illumination microscopy. Nat. Biotechnol., 36, 451-459(2018).

[11] A. Markwirth, M. Lachetta, V. Mönkemöller, R. Heintzmann, W. Hübner, T. Huser, M. Müller. Video-rate multi-color structured illumination microscopy with simultaneous real-time reconstruction. Nat. Commun., 10, 4315(2019).

[12] A. Sandmeyer, M. Lachetta, H. Sandmeyer, W. Hübner, T. Huser, M. Müller. DMD-based super-resolution structured illumination microscopy visualizes live cell dynamics at high speed and low cost(2019).

[13] M. G. Gustafsson, L. Shao, P. M. Carlton, C. R. Wang, I. N. Golubovskaya, W. Z. Cande, D. A. Agard, J. W. Sedat. Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination. Biophys. J., 94, 4957-4970(2008).

[14] M. Müller, V. Mönkemöller, S. Hennig, W. Hübner, T. Huser. Open-source image reconstruction of super-resolution structured illumination microscopy data in ImageJ. Nat. Commun., 7, 10980(2016).

[15] A. Lal, C. Shan, P. Xi. Structured illumination microscopy image reconstruction algorithm. IEEE J. Sel. Top. Quantum Electron., 22, 50-63(2016).

[16] P. Křžek, T. Lukeš, M. Ovesný, K. Fliegel, G. M. Hagen. SIMToolbox: a MATLAB toolbox for structured illumination fluorescence microscopy. Bioinformatics, 32, 318-320(2016).

[17] K. Wicker, O. Mandula, G. Best, R. Fiolka, R. Heintzmann. Phase optimisation for structured illumination microscopy. Opt. Express, 21, 2032-2049(2013).

[18] J. Fan, X. Huang, L. Li, S. Tan, L. Chen. A protocol for structured illumination microscopy with minimal reconstruction artifacts. Biophys. Rep., 5, 80-90(2019).

[19] D. P. Hoffman, E. Betzig. Tiled reconstruction improves structured illumination microscopy(2020).

[20] L. Jin, B. Liu, F. Zhao, S. Hahn, B. Dong, R. Song, T. C. Elston, Y. Xu, K. M. Hahn. Deep learning enables structured illumination microscopy with low light levels and enhanced speed. Nat. Commun., 11, 1934(2020).

[21] C. N. Christensen, E. N. Ward, P. Lio, C. F. Kaminski. ML-SIM: a deep neural network for reconstruction of structured illumination microscopy images(2020).

[22] C. Ling, C. Zhang, M. Wang, F. Meng, L. Du, X. Yuan. Fast structured illumination microscopy via deep learning. Photon. Res., 8, 1350-1359(2020).

[23] M. Weigert, U. Schmidt, T. Boothe, A. Müller, A. Dibrov, A. Jain, B. Wilhelm, D. Schmidt, C. Broaddus, S. Culley, M. Rocha-Martins, F. Segovia-Miranda, C. Norden, R. Henriques, M. Zerial, M. Solimena, J. Rink, P. Tomancak, L. Royer, F. Jug, E. W. Myers. Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat. Methods, 15, 1090-1097(2018).

[24] X. Mao, C. Shen, Y.-B. Yang. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. Advances in Neural Information Processing Systems, 2802-2810(2016).

[25] D. Sage, H. Kirshner, T. Pengo, N. Stuurman, J. Min, S. Manley, M. Unser. Quantitative evaluation of software packages for single-molecule localization microscopy. Nat. Methods, 12, 717-724(2015).

[26] T. Novák, T. Gajdos, J. Sinkó, G. Szabó, M. Erdélyi. TestSTORM: versatile simulator software for multimodal super-resolution localization fluorescence microscopy. Sci. Rep., 7, 951(2017).

[27] B. Lim, S. Son, H. Kim, S. Nah, K. Lee. Enhanced deep residual networks for single image super-resolution. IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 1132-1140(2017).

[28] Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, Y. Fu. Image super-resolution using very deep residual channel attention networks. European Conference on Computer Vision (ECCV), 286-301(2018).

[29] C. Karras, M. Smedh, R. Förster, H. Deschout, J. Fernandez-Rodriguez, R. Heintzmann. Successful optimization of reconstruction parameters in structured illumination microscopy–a practical guide. Opt. Commun., 436, 69-75(2019).

[30] O. Ronneberger, P. Fischer, T. Brox. U-Net: convolutional networks for biomedical image segmentation(2015).

[31] J. F. Abascal, S. Bussod, N. Ducros, S. Si-Mohamed, P. Douek, C. Chappard, F. Peyrin. A residual U-Net network with image prior for 3D image denoising(2020).

[32] A. Hore, D. Ziou. Image quality metrics: PSNR vs. SSIM. 20th International Conference on Pattern Recognition, 2366-2369(2010).

[33] L. Torrey, J. Shavlik. Transfer learning. Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods, and Techniques, 242-264(2010).

[34] J. Howard, S. Ruder. Universal language model fine-tuning for text classification. 56th Annual Meeting of the Association for Computational Linguistics, 328-339(2018).

[35] K. Dabov, A. Foi, V. Katkovnik, K. Egiazarian. Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process., 16, 2080-2095(2007).

[36] J. Lehtinen, J. Munkberg, J. Hasselgren, S. Laine, T. Karras, M. Aittala, T. Aila. Noise2Noise: learning image restoration without clean data. 35th International Conference on Machine Learning, 2965-2974(2018).

[37] J.-Y. Zhu, T. Park, P. Isola, A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. IEEE International Conference on Computer Vision, 2223-2232(2017).

[38] M. Van Heel, M. Schatz. Fourier shell correlation threshold criteria. J. Struct. Biol., 151, 250-262(2005).

[39] R. P. Nieuwenhuizen, K. A. Lidke, M. Bates, D. L. Puig, D. Grünwald, S. Stallinga, B. Rieger. Measuring image resolution in optical nanoscopy. Nat. Methods, 10, 557-562(2013).

[40] A. C. Descloux, K. S. Grussmayer, A. Radenovic. Parameter-free image resolution estimation based on decorrelation analysis. Nat. Methods, 16, 918-924(2019).

[41] R. Van den Eynde, A. Sandmeyer, W. Vandenberg, S. Duwé, W. Hübner, T. Huser, P. Dedecker, M. Müller. Quantitative comparison of camera technologies for cost-effective super-resolution optical fluctuation imaging (SOFI). J. Phys. Photon., 1, 044001(2019).

CLP Journals