
- Advanced Photonics Nexus
- Vol. 2, Issue 4, 046005 (2023)
Abstract
Keywords
1 Introduction
Utilizing spatial frequency modulation, structured illumination microscopy (SIM) transfers the normally inaccessible high-frequency information into measured low-frequency images with moiré effect, and thus it can extract the superresolution image by spatial frequency recombination, which breaks through the diffraction limit barrier of the optical microscope.1 Featuring high imaging speed, low excitation power, and high compatibility for various fluorescent labels, SIM has been widely applied in the superresolution imaging of biological specimens, especially the fine structure of organelles, such as actin cytoskeleton,2 mitochondria,3 lysosomes,4 and endoplasmic reticulum,5,6 and also the dynamics of organelles, such as mitochondrial fission,7 internalization of clathrin-coated pits (CCPs),8 and tubulin and kinesin dynamics.9 However, the resolution of SIM with linear illumination mode is only 2 times over that of wide-field microscopy, which reaches about 100 nm, because the maximal frequency of structured illumination pattern is limited by the diffraction. Thus, SIM is inferior to some other superresolution techniques in spatial resolution, such as stimulated emission depletion microscopy,10 photoactivated localization microscopy,11 or stochastic optical reconstruction microscopy,12 which can obtain the resolution up to about 10 nm. By now, many methods have been reported to increase the spatial resolution of SIM. For example, fluorescence saturation was utilized to induce nonlinear structured illumination for higher spatial frequency shift.13 Similarly, photoswitching proteins were used as the fluorescent labels for nonlinear SIM with low excitation power.14 With these methods, the nonlinear SIM with a resolution below 50 nm has been realized, but unfortunately it comes at the expense of reducing the imaging speed. In addition, a sparse deconvolution method (sparse-SIM) was proposed to further obtain the resolution enhancement of SIM while keeping the imaging speed by employing the sparsity and continuity priors of biological structures.15 Without the modification to the hardware device or acquirement mode of SIM, sparse-SIM achieves almost twofold resolution enhancement compared with conventional SIM. However, as an iterative deconvolution method, sparse-SIM is greatly affected by the artificial selection of multiple parameters, which are used to balance the weights of various priors. Therefore, the image processing in sparse-SIM is time-consuming.
In the experimental acquisition of a fluorescent image, strong background and noise are common issues. These issues may be tolerated in normal microscopy analysis but can lead to severe artifacts in the superresolution image reconstruction, which make it difficult to distinguish the real structures and fake artifacts. The strong background mainly results from the out-of-focus fluorescence and cellular autofluorescence, while the strong noise often occurs in the image acquirement by the camera, especially when the fluorescent photon number is limited by the short exposure time or low excitation power. To further improve the resolution of SIM under strong background and noise levels, here we propose a deep resolution enhancement (DRE) method by an untrained neural network associated with alternating direction method of multipliers (ADMM) framework, termed as ADMM-DRE-SIM. An untrained neural network with implicit priors is utilized as a generator for desired resolution-enhanced image, and an image degradation model containing optical transfer of SIM and additional background map provides the criterion for updating the parameters of the neural network. The untrained neural network is quite convenient to obtain the resolution enhancement of SIM because no training datasets are required (it is rather difficult to experimentally acquire the training datasets containing the SIM images and paired resolution-enhanced images). An iterative framework of ADMM embedded with Hessian prior is used to optimize the network by exploiting the intrinsic continuity of biological structures. Based on these special designs, ADMM-DRE-SIM realizes the resolution enhancement of SIM under the strong background and noise while completely preserving the advantage of SIM with high imaging speed. Compared with conventional SIM, a resolution enhancement by a factor of about 1.6 is experimentally demonstrated by imaging tubulins and actins in cells with ADMM-DRE-SIM. Moreover, ADMM-DRE-SIM shows better performance in the fidelity of biological structure compared to other deconvolution methods, which is key for image analysis, and it will provide a powerful tool for the studies of organelle dynamics.
2 Methods
The resolution of the image acquired in wide-field microscopy is restricted by the optical diffraction limit due to the finite aperture of the objective. To exceed the resolution limitation, a structured illumination pattern is utilized to modulate the sample image with spatial frequency shift in SIM. The recorded image in SIM can be formatted in the frequency domain as
Sign up for Advanced Photonics Nexus TOC. Get the latest issue of Advanced Photonics Nexus delivered right to you!Sign up now
To further improve the resolution of SIM under the strong background and noise levels, we adopt a concept of deep image prior (DIP), which indicates that the structure of a generator network itself can capture lots of statistics priors of low-level image for various tasks instead of learning from training datasets.19 The untrained neural network incorporated with various physical models has shown to be a well-established tool in solving the image reconstruction of computational imaging methods, such as coherent phase imaging,20 digital in-line holography,21,22 ghost imaging,23 and SIM.24,25 Here we achieve the further resolution enhancement of SIM by the untrained neural network combining with the background interrupted optical transfer model of SIM. The implicit priors inside the neural network provide the bridge between the available information in the SIM image and the higher frequency information corresponding to the higher resolution. Meanwhile, the optical transfer model containing the diffraction limit in SIM and the background map is employed to supervise the neural network and guarantee the image fidelity with background suppression. Furthermore, an ADMM framework embedded with Hessian prior is used to optimize the parameters of the neural network with high precision, which is different from the previous work with only an untrained neural network.26
The flow chart of ADMM-DRE-SIM is shown in Fig. 1. The original SIM image
Figure 1.Flow chart of ADMM-DRE-SIM for realizing the resolution enhancement of SIM.
By further introducing the auxiliary variable
The subproblem in Eq. (8) is similar to the traditional DIP solution by forcing
3 Results and Discussion
To demonstrate the resolution enhancement of SIM by ADMM-DRE-SIM, a theoretical simulation is conducted. A simulated image containing the curves with width of 32.5 nm is used as the ground truth. The wide-field image is generated by the convolution of the ground truth and the PSF of a microscope with the NA of 1.2 and fluorescence wavelength of 560 nm. Nine images are simulated by sequential structured illumination and PSF convolution based on the ground truth. A fairSIM algorithm is utilized to recover the superresolution SIM image.29 Then, the SIM image is further processed by ADMM-DRE-SIM to obtain a resolution-enhanced image. All the image processing is performed on a server with Intel i9-10920X, 192 GB of RAM and NVIDIA GeForce RTX3090. The simulation results are shown in Fig. 2, and the corresponding spatial frequency spectra and the intensity profiles along the labeled lines are also provided. As can be seen, only the low-frequency components are preserved in the wide-field image due to the limited OTF of the microscope, which lead to low spatial resolution, and the curves along the labeled line are seriously blurred. The SIM image has a frequency bound twice that of the wide-field image, and the intensity profile of the curves can be distinguished with difficulty. However, the ADMM-DRE-SIM image has a greatly extended frequency bound, which is close to the ground truth, and the adjacent curves can be clearly distinguished.
Figure 2.Simulation results for the resolution enhancement of SIM by ADMM-DRE-SIM. (a) The ground truth, (b) wide-field, (c) SIM, and (d) ADMM-DRE-SIM images of the simulated structures, associated with the corresponding frequency spectra and intensity distributions along the labeled lines in the insets. The insets are the enlarged views of the selected areas with yellow squares.
In actual experiments, the background and noise are inevitable, and thus the recorded images with structured illumination in SIM will be contaminated. Although SIM can suppress the background to some extent, heavy noise will cause vague artifacts during SIM reconstruction. As for the resolution improvement in noisy conditions, the noise will mislead the optimization direction in iterative calculations. The heavier the noise is, the more effort the algorithm takes to cover the noise. This leads to a bias between the reconstructed result and the real structure, which finally results in serious artifacts. Besides, the more resolution enhancement is conducted, the more sensitive to noise the reconstructed result is. The SIM images involving background and various noise levels are processed to test the resolution enhancement performance by ADMM-DRE-SIM. The simulated results are shown in Fig. 3; here, a uniform background combined with Gaussian noise
Figure 3.Simulation results for the effects of background and noise levels on the resolution enhancement of SIM by various algorithms. (a) The ground truth, wide-field, SIM, and ADMM-DRE-SIM images of the simulated structures without the background and noise. (b) The SIM, Wiener deconvolution, RLTV deconvolution, Hessian deconvolution, and ADMM-DRE-SIM images with a uniform background combined with Gaussian noise
Furthermore, two experiments are performed by imaging the intracellular structures with a commercial SIM system (Nikon, N-SIM) to further demonstrate the resolution enhancement capability of ADMM-DRE-SIM. The exposure time is set as 200 ms with illumination power of about 22.9 mW. First, the tubulins in mouse embryonic fibroblast (MEF) cells are investigated, as shown in Fig. 4(a). Tubulin is part of the cytoskeleton of cells and has a filamentous distribution with large space. Compared to SIM, ADMM-DRE-SIM provides finer structural details of the tubulins, and the overlapped fibers can be distinguished more clearly. Meanwhile, the intensity profiles along the labeled line also intuitively show the resolution enhancement. To quantitatively determine the resolution enhancement, a Fourier ring correlation (FRC) method is utilized to characterize the spatial resolution, and the calculated results are given in Fig. 4(b). The resolution is improved from 145 nm of SIM to 82 nm by ADMM-DRE-SIM, which indicates that ADMM-DRE-SIM is an effective technique to improve the resolution of SIM. Second, the actins in NIH/3T3 cells are further investigated, as shown in Fig. 4(c). Different from the tubulins, the distribution of actins in cells is much denser, and the fiber is thinner. Thus, the images of actins show a stronger background due to the out of focus fluorescence, which bring the difficulty for further resolution enhancement. Similarly, ADMM-DRE-SIM shows clearer spatial structure of actins compared with SIM. Importantly, ADMM-DRE-SIM can greatly suppress the artifacts of the SIM image due to the calculation error of the background in the image reconstruction. According to the calculated results of FRC in Fig. 4(d), the resolution of ADMM-DRE-SIM is enhanced to 85 nm from 140 nm of SIM. Based on our experimental observations in Fig. 4, ADMM-DRE-SIM can realize the resolution enhancement by a factor of about 1.6.
Figure 4.Experimental results for the resolution enhancement of SIM by ADMM-DRE-SIM. (a) The wide-field, SIM, and ADMM-DRE-SIM images of tubulins in MEF cells and (c) actins in NIH/3T3 cells, together with the corresponding intensity distributions along the labeled lines in the insets. The FRCs of the SIM and ADMM-DRE-SIM images for (b) tubulins and (d) actins.
To specifically exhibit the superiority of ADMM-DRE-SIM in suppressing the background and noise, the SIM images of tubulins with strong background or low signal-to-noise ratio (SNR) are separately collected and processed in experiment. As shown in Fig. 5(a), the tubulin fibers in the SIM image are seriously contaminated by the background, although SIM has the function of optical sectioning.33 The strong background results from imperfect fluorescence staining during sample preparation. The resolution-enhanced images by conventional deconvolution algorithms of RL, RLTV, and Hessian are all affected by the background and therefore contain severe artifacts. On the contrary, ADMM-DRE-SIM can eliminate the background and keep the fiber structures with enhanced resolution because the image degradation model in ADMM-DRE-SIM has separated the background. And the sparse deconvolution has similar performance. As shown in Fig. 5(b), the SIM image has quite a low SNR due to the recorded nine images, with structured illumination being acquired at high speed with a short exposure time (0.5 ms),4 which results in a great challenge for further resolution enhancement. In the same way, the noise is greatly amplified using conventional deconvolution algorithms to process the SIM image with low SNR, and the artifacts are very serious and almost confused with the tubulin fibers. In contrast, ADMM-DRE-SIM succeeds in suppressing the artifacts and acquiring a resolution-enhanced image with high-fidelity structures.
Figure 5.Experimental results for the effects of the strong background and low SNR on the resolution enhancement of SIM by various algorithms. (a) SIM image of tubulins with strong background. Left, the stitched image with the SIM image on the top and the ADMM-DRE-SIM image on the bottom; right, the enlarged SIM image marked with the yellow square and the images processed by RL, RLTV, Hessian, sparse deconvolution, and ADMM-DRE-SIM algorithms, respectively. (b) SIM image of tubulins with low SNR. The image arrangement is the same as (a).
To verify the superiority of ADMM-DRE-SIM in structural fidelity, the SIM image of tubulins with strong background6 is processed by ADMM-DRE-SIM and modified ADMM-DRE-SIM; here the modified ADMM-DRE-SIM means that the background estimation is directly subtracted from the input SIM image, and the image degradation model has only the convolution of PSF of SIM. The SIM image in Fig. 6(a) has the strong background, and the background map can be estimated by the wavelet transform, as shown in Fig. 6(b). The background subtracted SIM image has obvious discontinuity, as shown in Fig. 6(c). As shown in Figs. 6(f) and 6(g), ADMM-DRE-SIM and the modified version both can achieve the resolution enhancement with sharper profiles compared with the SIM image. The direct subtraction of the estimated background map leads to the accidental removal of the useful structures, as shown in Fig. 6(e) and therefore it is very difficult to recover the missing structures during the image processing of resolution enhancement without the additional information, as shown in Fig. 6(f). However, ADMM-DRE-SIM addresses this issue by inserting the background estimation into the image degradation model instead of the direct subtraction, where the accidental removal of the structures can be eliminated during the iterative optimization of the network for the resolution enhancement. The resolution-enhanced image in Fig. 6(g) by ADMM-DRE-SIM shows the higher fidelity with fewer missing structures compared with the image processed by modified ADMM-DRE-SIM with direct subtraction of background estimation in Fig. 6(f). By comparison, it is obvious that ADMM-DRE-SIM has the stronger robustness and can preserve the structural information while removing the background and noise.
Figure 6.Comparison of ADMM-DRE-SIM and modified ADMM-DRE-SIM in structural fidelity. (a) SIM image of tubulins; (b) estimated background map; (c) image by directly subtracting the estimated background map from the SIM image. (d)–(g) The SIM, background subtracted SIM, modified ADMM-DRE-SIM, and ADMM-DRE-SIM images for the selected area marked with the yellow square, respectively.
For the superresolution imaging of small intracellular structures, the pixel size of the camera is also an important factor for the final imaging performance. The smaller pixel size can contribute to the higher resolution bound, while it also leads to the lower SNR, and therefore the pixel size of the camera used in superresolution imaging is usually a few micrometers. However, the pixel effect sometimes results in a discrete profile of the superresolution structures. To acquire a resolution-enhanced image of the finer structures with smooth profile, a
Figure 7.Experimental results for the resolution enhancement of SIM by ADMM-DRE-SIM with
As shown above, ADMM-DRE-SIM shows a powerful ability to transfer the resolution-limited SIM image to the resolution-enhanced image by bridging the low and high frequencies with the implicit priors from the network and Hessian continuity. Meanwhile, an inner image degradation model containing the equivalent PSF of SIM and background map offers the resistance to the strong background. It should be pointed out that the image degradation model is utilized in each iteration during the optimization to maintain the structural integrity because a simple subtraction of the background estimation may lead to the structural defects, especially under the low SNR condition. ADMM-DRE-SIM not only realizes the resolution enhancement of SIM but also retains the advantages of SIM involving high imaging speed, low photo-damage, and wide applications for various fluorescent labels. Therefore, ADMM-DRE-SIM provides a well-established tool to capture the high-speed dynamics of finer structures, which will have important applications in many biological areas, such as organelle dynamics34 and cell metabolism.35 However, ADMM-DRE-SIM has also some limitations. The imaging degradation model is based on the premise of uniform PSF across the whole image, which is an ideal situation and not very consistent with actual experiments; thus the bias may induce uneven resolution enhancement. This issue can be addressed by employing space-variation PSF. In addition, the SIM images in this work are reconstructed by fairSIM, which is a classic algorithm without the priors, and so the effect of resolution enhancement by ADMM-DRE-SIM is obvious due to the use of implicit priors. If the SIM images are reconstructed by TVSIM36 or Hessian-SIM,4 which have already exploited the priors, the resolution enhancement ability of ADMM-DRE-SIM will be slightly reduced.
4 Conclusions
In summary, we have developed an ADMM-DRE-SIM to further improve the resolution of SIM by an untrained neural network optimized with an ADMM framework. By incorporating an image degradation model containing equivalent PSF of SIM and additional background map, ADMM-DRE-SIM can significantly improve the resolution of SIM even under the condition of strong background and noise levels. A resolution enhancement by a factor of about 1.6 was experimentally demonstrated by the superresolution imaging of tubulins and actins in cells. Possessing the resolution enhancement ability while maintaining the intrinsic advantages of SIM, ADMM-DRE-SIM can provide a flexible and robust tool for studying the high-speed dynamics of biological fine structures, which will greatly promote the development of biomedical imaging. Furthermore, by inserting the imaging degradation model of three-dimensional (3D) SIM37 into the framework of ADMM-DRE-SIM, 3D resolution enhancement can also be achieved. Additionally, ADMM-DRE-SIM is not limited to the resolution enhancement of SIM. The superresolution microscopy images acquired by some other techniques can also be further processed to obtain the higher resolution once the corresponding equivalent PSF is provided, for example, confocal spinning-disk,38 two-photon,39 and expansion microscopy.40
Yu He is a PhD student studying at the State Key Laboratory of Precision Spectroscopy, East China Normal University (ECNU) under the supervisions of Prof. Shian Zhang. His research focuses on high-speed super-resolution microscopy.
Yunhua Yao is an associate professor from the State Key Laboratory of Precision Spectroscopy at ECNU. He obtained his PhD in Optics from ECNU in 2018. His current research interest focuses on high-speed super-resolution microscopy and ultrafast optical imaging.
Yilin He is a PhD student studying at the State Key Laboratory of Precision Spectroscopy, ECNU under the supervisions of Prof. Shian Zhang. His research focuses on high-speed super-resolution microscopy.
Zhengqi Huang is a PhD student studying at the State Key Laboratory of Precision Spectroscopy, ECNU under the supervisions of Prof. Shian Zhang. His research focuses on high-speed super-resolution microscopy.
Dalong Qi is a young professor from the State Key Laboratory of Precision Spectroscopy at ECNU. He obtained his PhD in Optics from ECNU in 2017. His current research interest focuses on ultrafast optical and electronic imaging techniques and their applications.
Chonglei Zhang is a professor from Nanophotonics Research Center at Shenzhen University. He obtained his PhD in Physics from Nankai University in 2007. His current research interest focuses on super resolution microscopy and surface plasmon resonance.
Xiaoshuai Huang received his bachelor’s degree in science from Wuhan University in 2013 and his PhD from Peking University in 2018. He is an assistant professor at Peking University. From 2018 to 2020, he was trained as a postdoctoral research fellow at Peking University. His research is focused on super-resolution microscopy and cell biology.
Kebin Shi is a professor from State Key Laboratory for Mesoscopic Physics at Peking University. He obtained his PhD from the Pennsylvania State University. His current research interest focuses on nonlinear photonics and biomedical imaging.
Pengpeng Ding is a PhD student studying at the State Key Laboratory of Precision Spectroscopy, ECNU under the supervisions of Prof. Shian Zhang. His research focuses on ultrafast optical imaging.
Chengzhi Jin is a PhD student studying at the State Key Laboratory of Precision Spectroscopy, ECNU under the supervisions of Prof. Shian Zhang. His research focuses on ultrafast optical imaging.
Lianzhong Deng is an associate professor from the State Key Laboratory of Precision Spectroscopy at ECNU. He obtained his PhD in Optics from ECNU in 2008. His current research interest focuses on light field manipulation and their applications.
Zhenrong Sun is a professor of the State Key Laboratory of Precision Spectroscopy at ECNU. He obtained his PhD in Physics from ECNU in 2007. His current research interest focuses on ultrafast dynamics of clusters and ultrafast optical imaging.
Xiaocong Yuan is a professor and the director of Nanophotonics Research Center at Shenzhen University. He obtained his PhD in Physics from King’s College London in 1994. His current research interest focuses on optical manipulation, high-sensitivity sensor, super-resolution microscopy and surface-enhanced Raman spectroscopy.
Shian Zhang is a professor and the deputy director of the State Key Laboratory of Precision Spectroscopy at ECNU. He obtained his PhD in Optics from ECNU in 2006. His current research interest focuses on ultrafast optical imaging, high-speed super-resolution microscopy and light field manipulation.
References
[16] J. B. Sibarita. Deconvolution microscopy. Adv. Biochem. Eng./Biotechnol., 95, 201-243(2005).
[19] D. Ulyanov, A. Vedaldi, V. Lempitsky. Deep image prior, 9446-9454(2018).
[20] F. Wang et al. Phase imaging with an untrained neural network. Light: Sci. Appl., 9, 77(2020).
[27] D. P. Kingma, J. Ba. Adam: a method for stochastic optimization(2014).

Set citation alerts for the article
Please enter your email address