• Advanced Photonics Nexus
  • Vol. 2, Issue 1, 016012 (2023)
Jianyong Wang1、2、†, Junchao Fan3, Bo Zhou1, Xiaoshuai Huang4、5、*, and Liangyi Chen1、6、7、8、*
Author Affiliations
  • 1Peking University, Institute of Molecular Medicine, College of Future Technology, Center for Life Sciences, State Key Laboratory of Membrane Biology, Beijing Key Laboratory of Cardiometabolic Molecular Medicine, Beijing, China
  • 2Peking University, School of Software and Microelectronics, Beijing, China
  • 3Chongqing University of Posts and Telecommunications, College of Computer Science and Technology, Chongqing Key Laboratory of Image Cognition, Chongqing, China
  • 4Peking University, Biomedical Engineering Department, Beijing, China
  • 5Peking University, International Cancer Institute, Beijing, China
  • 6PKU-IDG/McGovern Institute for Brain Research, Beijing, China
  • 7Beijing Academy of Artificial Intelligence, Beijing, China
  • 8National Biomedical Imaging Center, Beijing, China
  • show less
    DOI: 10.1117/1.APN.2.1.016012 Cite this Article Set citation alerts
    Jianyong Wang, Junchao Fan, Bo Zhou, Xiaoshuai Huang, Liangyi Chen. Hybrid reconstruction of the physical model with the deep learning that improves structured illumination microscopy[J]. Advanced Photonics Nexus, 2023, 2(1): 016012 Copy Citation Text show less
    TDV-SIM diagrams and parameter selection. (a) TDV-SIM reconstruction pipeline. (b) Visualization of TDV and its gradient. Mi is a residual structure micro-block and GM is its gradient. Conv is the convolution layer and T-Conv is its gradient. Act is the activation layer and GAct is its gradient. (c) Actin filaments SIM SR image. (d) Top, PSNR and SSIM of TDV-SIM reconstructions with different λ and T; bottom, TDV-SIM reconstruction durations of single SR image (1024×1024) with different T (n=5). (e) Magnified views of the boxed regions in panel (c). Yellow arrowheads highlight artifacts not eliminated with too small λ or T. White arrowheads indicate incorrectly removed signals with too large λ or T. Scale bars: (c) 1 μm and (e) 0.5 μm.
    Fig. 1. TDV-SIM diagrams and parameter selection. (a) TDV-SIM reconstruction pipeline. (b) Visualization of TDV and its gradient. Mi is a residual structure micro-block and GM is its gradient. Conv is the convolution layer and T-Conv is its gradient. Act is the activation layer and GAct is its gradient. (c) Actin filaments SIM SR image. (d) Top, PSNR and SSIM of TDV-SIM reconstructions with different λ and T; bottom, TDV-SIM reconstruction durations of single SR image (1024×1024) with different T (n=5). (e) Magnified views of the boxed regions in panel (c). Yellow arrowheads highlight artifacts not eliminated with too small λ or T. White arrowheads indicate incorrectly removed signals with too large λ or T. Scale bars: (c) 1  μm and (e) 0.5  μm.
    TDV-SIM outperforms other reconstruction algorithms in suppressing artifacts and hallucinations while maintaining resolution. (a) Actin filaments under the SR-SIM. (b) Magnified views of the larger boxed region in panel (a) reconstructed by Wiener deconvolution, HiFi-SIM, Hessian-SIM, and TDV-SIM. The GT image is shown as the reference. Profiles along the yellow line are on the bottom. (c) Magnified views of the smaller boxed regions in panel (a) reconstructed by Wiener deconvolution, scU-Net, DFCAN, and TDV-SIM. The GT images are shown as references. (d) Time series imaging of ER under the SR-SIM (Video 1, MP4, 45 MB [URL: https://doi.org/10.1117/1.APN.2.1.016012.s1]). (e) Magnified views of the boxed regions in panel (d) reconstructed by Wiener deconvolution, HiFi-SIM, Hessian-SIM, scU-Net, DFCAN, and TDV-SIM. The GT images are shown as references. Artifact variances of actin filaments (f) or ER tubules (g) from background regions in different reconstructions (n=15 from three cells for each sample). Yellow arrowheads in panels (c) and (e) indicate the inaccurate reconstructions of pure DL-based methods. Red arrowheads in panel (e) highlight the artifacts of physical-model-based methods. SSIM of actin filaments (h) and ER tubules (i) in different reconstructions (n=150 and 15, respectively). (j) Resolutions of different reconstructions of actin filaments in panels (a)–(c) (n=14 from three cells). Scale bars: (a) and (d) 1 μm; (b) top, (c) and (e) 0.5 μm. (b) Bottom, axial: 0.2 arbitrary units (a.u.); lateral: 0.1 μm. Data are shown as mean ± SEM. p<0.05*, p<0.01**, p<0.001***, ns: not significant (one-way ANOVA).
    Fig. 2. TDV-SIM outperforms other reconstruction algorithms in suppressing artifacts and hallucinations while maintaining resolution. (a) Actin filaments under the SR-SIM. (b) Magnified views of the larger boxed region in panel (a) reconstructed by Wiener deconvolution, HiFi-SIM, Hessian-SIM, and TDV-SIM. The GT image is shown as the reference. Profiles along the yellow line are on the bottom. (c) Magnified views of the smaller boxed regions in panel (a) reconstructed by Wiener deconvolution, scU-Net, DFCAN, and TDV-SIM. The GT images are shown as references. (d) Time series imaging of ER under the SR-SIM (Video 1, MP4, 45 MB [URL: https://doi.org/10.1117/1.APN.2.1.016012.s1]). (e) Magnified views of the boxed regions in panel (d) reconstructed by Wiener deconvolution, HiFi-SIM, Hessian-SIM, scU-Net, DFCAN, and TDV-SIM. The GT images are shown as references. Artifact variances of actin filaments (f) or ER tubules (g) from background regions in different reconstructions (n=15 from three cells for each sample). Yellow arrowheads in panels (c) and (e) indicate the inaccurate reconstructions of pure DL-based methods. Red arrowheads in panel (e) highlight the artifacts of physical-model-based methods. SSIM of actin filaments (h) and ER tubules (i) in different reconstructions (n=150 and 15, respectively). (j) Resolutions of different reconstructions of actin filaments in panels (a)–(c) (n=14 from three cells). Scale bars: (a) and (d) 1  μm; (b) top, (c) and (e) 0.5  μm. (b) Bottom, axial: 0.2 arbitrary units (a.u.); lateral: 0.1  μm. Data are shown as mean ± SEM. p<0.05*, p<0.01**, p<0.001***, ns: not significant (one-way ANOVA).
    TDV-SIM enables accurate reconstruction of intricate and dynamic mitochondrial cristae structures in live cells after prolonged bleaching. (a) Mitochondria under the SR-SIM. (b) Time-dependent bleaching in fluorescence intensities of mitochondria. (c) Magnified views of the larger boxed region in panel (a) reconstructed by scU-Net, DFCAN, and TDV-SIM and the corresponding GT image at 0 s. Profiles along the blue line are on the right. (d) Magnified views of the smaller boxed region in panel (a) reconstructed by Wiener deconvolution, HiFi-SIM, Hessian-SIM, and TDV-SIM and the corresponding GT images at 0, 15, and 20 s. (e) The SSIMs of regions enclosed mitochondria from different reconstructions compared to GT images at 0, 15, and 20 s (n=15). (f) Artifact variances of the background regions in different reconstructions at 0, 15, and 20 s (n=15). Scale bars: (a) 1 μm; (c) and (d) 0.5 μm. Data are shown as mean ± SEM. p<0.05*, p<0.01**, p<0.001***, ns: not significant (one-way ANOVA).
    Fig. 3. TDV-SIM enables accurate reconstruction of intricate and dynamic mitochondrial cristae structures in live cells after prolonged bleaching. (a) Mitochondria under the SR-SIM. (b) Time-dependent bleaching in fluorescence intensities of mitochondria. (c) Magnified views of the larger boxed region in panel (a) reconstructed by scU-Net, DFCAN, and TDV-SIM and the corresponding GT image at 0 s. Profiles along the blue line are on the right. (d) Magnified views of the smaller boxed region in panel (a) reconstructed by Wiener deconvolution, HiFi-SIM, Hessian-SIM, and TDV-SIM and the corresponding GT images at 0, 15, and 20 s. (e) The SSIMs of regions enclosed mitochondria from different reconstructions compared to GT images at 0, 15, and 20 s (n=15). (f) Artifact variances of the background regions in different reconstructions at 0, 15, and 20 s (n=15). Scale bars: (a) 1  μm; (c) and (d) 0.5  μm. Data are shown as mean ± SEM. p<0.05*, p<0.01**, p<0.001***, ns: not significant (one-way ANOVA).
    TDV-SIM enables better reconstruction of actin filaments under NL-SIM. (a) Actin filaments under the NL-SIM. (b), (c) Magnified views of the white boxed regions in panel (a) reconstructed by Wiener deconvolution, Hessian-SIM, DFCAN, and TDV-NL-SIM. The GT image is shown as the reference. Profiles along the yellow line are on the bottom. (d) Magnified views of the yellow boxed regions in panel (a) reconstructed by Wiener deconvolution, Hessian-SIM, DFCAN, and TDV-NL-SIM. The GT image is shown as the reference. Yellow arrowheads indicate the inaccurate reconstructions of pure DL-based methods. (e) Artifact variances of actin filaments from background regions in different reconstructions (n=20). (f) Signal variance along the actin filaments in different reconstructions (n=20). (g) SSIM of actin filaments in different reconstructions (n=20). Scale bars: (a) 1 μm; (b), (c) top and (d) 0.5 μm. (b), (c) Bottom, axial: 0.2 arbitrary units (a.u.); lateral: 0.1 μm. Data are shown as mean ± SEM. p<0.01**, p<0.001***, ns: not significant (one-way ANOVA).
    Fig. 4. TDV-SIM enables better reconstruction of actin filaments under NL-SIM. (a) Actin filaments under the NL-SIM. (b), (c) Magnified views of the white boxed regions in panel (a) reconstructed by Wiener deconvolution, Hessian-SIM, DFCAN, and TDV-NL-SIM. The GT image is shown as the reference. Profiles along the yellow line are on the bottom. (d) Magnified views of the yellow boxed regions in panel (a) reconstructed by Wiener deconvolution, Hessian-SIM, DFCAN, and TDV-NL-SIM. The GT image is shown as the reference. Yellow arrowheads indicate the inaccurate reconstructions of pure DL-based methods. (e) Artifact variances of actin filaments from background regions in different reconstructions (n=20). (f) Signal variance along the actin filaments in different reconstructions (n=20). (g) SSIM of actin filaments in different reconstructions (n=20). Scale bars: (a) 1  μm; (b), (c) top and (d) 0.5  μm. (b), (c) Bottom, axial: 0.2 arbitrary units (a.u.); lateral: 0.1  μm. Data are shown as mean ± SEM. p<0.01**, p<0.001***, ns: not significant (one-way ANOVA).
    Jianyong Wang, Junchao Fan, Bo Zhou, Xiaoshuai Huang, Liangyi Chen. Hybrid reconstruction of the physical model with the deep learning that improves structured illumination microscopy[J]. Advanced Photonics Nexus, 2023, 2(1): 016012
    Download Citation