Superresolution (SR) fluorescence microscopy provides nanoscale resolution for studying subcellular structures and biological processes.1^–7 However, the higher light dose required for SR imaging than conventional microscopy, phototoxicity, and photobleaching severely limit their applications in live-cell imaging.8 Structured illumination microscopy (SIM) demonstrates a higher photon efficiency than other SR microscopy.9^–11 In particular, two-dimensinal (2D)-SIM can achieve a doubling resolution beyond the light diffraction limit using nine sequentially acquired images, making it useful for live-cell SR imaging. However, live-cell SR-SIM imaging still suffers from phototoxicity and photobleaching, and image restoration is an ill-posed inverse problem.9^,10^,12^–15 Therefore, for raw images of low signal-to-noise ratio (SNR) caused by short exposure or excessive photobleaching, the conventional Wiener-based reconstruction method is prone to artifacts.16^–19

Various physical model-based restoration methods have been developed to suppress SIM artifacts, such as total variation regularization,19 notch filtering,16^,18 high-fidelity (HiFi)-SIM,20 and joint space and frequency reconstruction-SIM.21 Using spatiotemporal continuity as the prior knowledge, we have developed an iterative restoration method based on the Hessian regularization term (Hessian-SIM) that suppresses artifacts due to the amplification of random noise.18 However, other artifacts persist, such as hammer-stroke and honeycomb artifacts due to the out-of-focus background17^,18^,20 and artifacts due to illumination scattering,22 which cannot be suppressed completely by model-based methods. Deep neural networks can approximate arbitrary functions with infinitesimal errors to extract high-dimensional features from low-resolution and low-quality images.23 Therefore, researchers have proposed end-to-end deep-learning (DL)-based reconstruction algorithms to suppress different artifacts indiscriminately with low SNR raw SIM images.24^–26 However, DL-based methods may suffer from hallucinations27 and generally reduced resolution. For example, current DL methods often incorrectly predict mitochondrial cristae structures in live cells.

To combine the advantages of both methods, we try to balance the reconstruction fidelity of traditional methods and the artifacts suppression of DL methods. However, suppose these two parts are combined into one objective function to achieve simultaneous optimization; in that case, the network is required to calculate the partial differential28 of the input images rather than the normal network weights. Therefore, we utilized the total deep variation (TDV) network as a regularizer in the reconstruction objective function. By combining the physical SIM reconstruction procedure with the TDV regularizer,28 we propose a hybrid restoration method (TDV-SIM) to suppress artifacts and maintain resolution simultaneously. On processing images of different cellular structures, TDV-SIM retains the actual signals better than the pure DL methods while removing artifacts more effectively than the model-based methods.

For SIM imaging, the sample is excited by sinusoidal illuminations with different pattern orientations and phases. The raw images contain low- and high-frequency information, which need to be separated and reassembled in SIM reconstruction.16^,18^,19 We transformed the SIM reconstruction into an optimization problem and constructed an objective function [Eq. (1)] composed of the fidelity term D(f,g) based on the physical model and the TDV regularization term R(f) based on DL (Supplementary Note 1 in the Supplementary Material), minfD(f,g)+λR(f),(1)where f is the target image to be estimated, g is the inverse Fourier transform of the high- and low-frequency information separated from the SIM raw data, and λ is the weight parameter of the regularization term. By optimizing the objective function with the gradient descent algorithm [Eq. (2)], TDV-SIM can reconstruct SR-SIM images that preserve the high-frequency information more faithfully than pure DL-based methods, and suppress artifacts more effectively than pure model-based methods, fk+1=fk−η∇D(fk,g)−ηλ∇R(fk),(2)where η is the step size. The entire reconstruction pipeline is shown in Fig. 1(a), where f0 is the initial SIM image obtained by Wiener deconvolution and fT is the final reconstruction after T iterations. The computation pipeline of ∇R(f) is shown in Fig. 1(b).

Compared to the ground truth (GT) image of actin filaments [averages of multiple Wiener-processed images, Fig. 1(c)], we quantized the peak SNR [PSNR; Fig. 1(d), top left] and structural similarity index measure [SSIM; Fig. 1(d), top right] values of TDV-SIM reconstructions with different weight parameters λ and iteration numbers T. The TDV-SIM reconstruction duration of a single SR image (1024×1024) with different iteration numbers T is also demonstrated [Fig. 1(d), bottom]. Through human inspection [Fig. 1(e)], artifacts may not be suppressed entirely if λ (or T) is too small; in contrast, if λ (or T) is too large with a fixed T of 25 (or a λ of 2.5), genuine signals may be removed incorrectly. Thus we set the optimal parameters to be 2.5 and 25 for λ and T, respectively.

COS-7 cells (ATCC and CRL-1651) were cultured in high-glucose Dulbecco’s modified Eagle’s medium (Gibco, 21063029) supplemented with 10% fetal bovine serum (Gibco) and 1% 100 mM sodium pyruvate solution (Sigma-Aldrich, S8636) in an incubator at 37 °C with 5% CO2 until reaching ∼75% confluency.

To label mitochondria, COS-7 cells were incubated with 250 nM MitoTracker Green FM (Thermo Fisher Scientific, M7514) in an hank’s balanced salt solution medium (Thermo Fisher Scientific, 14025076) containing Ca2+ and Mg2+ at 37 °C for 15 min, followed by washing 3 times before conducting 2D-SIM imaging. To label actin, COS-7 cells were transfected with Lifeact-enhanced green fluorescent protein (EGFP). According to the manufacturer’s instructions, the transfections were executed using Lipofectamine 2000 (Thermo Fisher Scientific, 11668019). After transfection, the cells were plated on precoated coverslips. Live cells were imaged in a complete cell culture medium containing no phenol red in a 37 °C live-cell imaging system. To label endoplasmic reticulum (ER), COS-7 cells were transfected with EGFP-Lys-Asp-Glu-Leu. According to the manufacturer’s instructions, the transfections were executed using Lipofectamine 3000 (Thermo Fisher Scientific, L3000015). After transfection, the cells were cultured for 20 to 28 h before the experiments. Live cells were imaged in a complete cell culture medium containing no phenol red in a 37 °C live-cell imaging system. The cells were tested for mycoplasma contamination before use.

The same SIM settings in Hessian-SIM18 were used. To obtain low SNR raw images and the corresponding GT images for training the neural network, we imaged the specimen with SIM. We recorded 20 images for each illumination pattern and then changed the phase and orientation of the pattern. We repeated the cycle nine times, corresponding to three orientations multiplied by three phases, thus obtaining 180 raw images. Then we divided the raw images into 20 groups, with each group containing nine illumination patterns of three phases and three orientations. After removing the fluorescent background, we can obtain 20 SR images with artifacts using Wiener deconvolution. Finally, we mimicked the artifact-free GT by averaging the 20 SR images.

We imaged ∼20cells, and the images were preprocessed to obtain pairs of raw data and GT images at each time point. Next, we divided such image pairs into a training set, a validation set, and a test set; then, we applied random cropping, quarter rotating, and horizontal/vertical flipping to further enrich the training data set. We trained the TDV-SIM using an Adam optimizer, with the learning rate set to 10−4. For actin, we adopted the mean square error (MSE) loss function, lMSE(X,Y)=1H×W∑i=1H∑j=1W(Xi,j−Yi,j)2,(3)where W and H represent the image width and height, respectively. For mitochondria and ER, a combination of the MSE loss and the SSIM loss was used, lcombination(X,Y)=lMSE(X,Y)+k[1−SSIM(X,Y)],(4)where k is a scalar weight that balances the relative contributions of SSIM and MSE losses and is set to 0.1 throughout this paper.

To avoid the influence of different methods on the dynamic range of the inferred SR images, we first normalize the SR images, Norm(X)=X−min(X)max(X)−min(X).(5)We used the PSNR, SSIM, and normalized root MSE (NRMSE) to evaluate the similarity between the reconstructed image and GT. They were calculated as follows: PSNR(X,Y)=10×lg[MAXI2∑i=1H∑j=1W(Xi,j−Yi,j)2/(H×W)],(6)SSIM(X,Y)=(2μXμY+c1)(2σXY+c2)(μX2+μY2+c1)(σX2+σY2+c2),(7)NRMSE(X,Y)=∑i=1H∑j=1W(Xi,j−Yi,j)2/(H×W)max(Y)−min(Y),(8)where W and H represent the image width and height, respectively. X and Y represent the reconstruction result and the GT image, respectively. MAXI is the maximum possible pixel value of the image and equals to 2B−1 when the image is represented with linear pulse-code modulation of B bits (e.g., MAXI equals 255 for an 8-bit image). μX and μY represent the averages of X and Y, σX and σY represent the variances of X and Y, and σXY represents the covariance of X and Y. c1 and c2 are small positive constants that stabilize each term; c1=(0.01L)2, c2=(0.03L)2, where L is the dynamic range of the pixel values.

Artifacts often emerged in regions of minor signals, such as the meshed region within actin filaments. Therefore, benchmarked against the GT, we selected these regions to calculate their variances.

We compared TDV-SIM with other reconstruction methods, including physical-model-based (Wiener deconvolution,11 HiFi-SIM, and Hessian-SIM) and pure DL-based methods [skip-layer connecting U-Nets (scU-Net)24 and deep Fourier channel attention network (DFCAN)25] using synthetic images with known GT (Fig. S1 in the Supplementary Material). TDV-SIM confers balanced performance in generating SR images of high SSIM, low NRMSE, and low artifacts among all reconstruction methods. Next, we examined dynamic actin filaments and ER in live cells observed with short exposures [actin: 1 ms, Fig. 2(a); 2.7 ms, Fig. S2a in the Supplementary Material; and ER: 0.789 ms, Fig. 2(d)]. Despite the improved reconstructions compared to the Wiener deconvolution, HiFi-SIM and Hessian-SIM still produced artifacts due to noise amplification in background regions with low SNR. TDV-SIM produced more continuous actin filaments (Fig. S2e in the Supplementary Material) with fewer artifacts but comparable SSIM values and resolutions to the conventional reconstruction methods [Figs. 2(b), 2(e), and 2(f)–2(j) and Fig. S2d in the Supplementary Material]. In contrast, pure DL-based methods led to reconstruction with fewer artifacts at the price of reduced resolution and decreased SSIM values. In addition, we often observed inaccurate inferences at the intersections of actin filaments and ER [yellow arrows in Figs. 2(c) and 2(e) and Fig. S2c in the Supplementary Material]. Together with the incorrectly inferred actin filaments at regions with extremely low fluorescence intensity (Fig. S3 in the Supplementary Material), these resembled the “hallucination effects” of pure DL methods,27 which was abolished by the TDV-SIM method. Furthermore, we compared TDV-SIM with rationalized DL (rDL) SIM29 on microtubule image from the BioSR25 data set (Fig. S4 in the Supplementary Material). By incorporating prior knowledge of illumination patterns into the DL network, rDL SIM aimed to denoise raw images rationally. Still, it produced punctated artifacts in background regions, which may be suppressed with a notch filter (NF) (white boxed region in Figs. S4(a) and S4(c) in the Supplementary Material). Moreover, we often observed microtubules within densely labeled regions absent from notch-filtered rDL SIM reconstructions (NF-rDL SIM; yellow arrows in Fig. S4b in the Supplementary Material), which was confirmed by the missing spikes in corresponding fluorescence profiles in the bottom. In comparison, TDV-SIM can avoid the missing signal problem of NF-rDL SIM and produce higher-fidelity reconstructions with fewer artifacts and higher SSIM [Figs. S4(b)–S4(d) in the Supplementary Material].

Photobleaching constitutes a major problem of fluorescence SR imaging, continuously reducing image SNR, and compromising the quality of reconstructed images, especially upon resolving nonstereotypical structures such as mitochondrial cristae.30 Therefore, we benchmarked the performance of TDV-SIM in resolving mitochondrial cristae dynamics for a prolonged time in live cells [Fig. 3(a)]. During the 20 s recording, the fluorescence intensity of MitoTracker decreased by ∼30% due to photobleaching [Fig. 3(b)]. In the beginning, model-based methods could reconstruct high-quality intricate mitochondrial cristae, which were gradually corrupted with artifacts gradually due to photobleaching [Figs. 3(d)–3(f)]. In contrast, although pure DL-based methods consistently generated fewer artifacts during the imaging period, they could not predict most cristae structures in the first place [Figs. 3(c), 3(e), and 3(f)]. Outperforming all other methods, TDV-SIM obtained sharp mitochondrial cristae structures with fewer artifacts and high SSIM with the GT, which persisted even under photobleaching conditions [Figs. 3(c)–3(f)].

In comparison to conventional linear SIM, nonlinear (NL) SIM achieves higher lateral resolution up to ∼60nm,9 whereas NL-SIM suffers from the reconstruction artifacts, especially with low SNR raw data. By combining the NL-SIM physical model with the TDV regularization term, we proposed the TDV-NL-SIM. We benchmarked the performance of TDV-NL-SIM with Wiener deconvolution, Hessian-NL-SIM, and DFCAN on actin filaments within the BioSR data set25 [Fig. 4(a)]. Similar to the linear SIM circumstances, Hessian-NL-SIM provided improved reconstructions than Wiener deconvolution but still produced significant artifacts in background regions. In contrast, TDV-NL-SIM produced more continuous actin filaments [Figs. 4(c) and 4(f)] with fewer artifacts but comparable SSIM values to Hessian-NL-SIM [Figs. 4(b), 4(e), and 4(g)]. DFCAN led to reconstruction with comparable continuity but decreased SSIM values to TDV-NL-SIM [Figs. 4(f) and 4(g)]. And the inaccurate inferences of DFCAN at the actin filaments intersections can be avoided by the TDV-NL-SIM [yellow arrows in Fig. 4(d)].

For traditional reconstruction methods such as Hessian-SIM, the denoising effect is limited to images with a low SNR. In contrast, the pure DL method directly fits the SR image through raw images, in which the fitting process is a black box. Therefore, reconstruction fidelity entirely depends on the network fitting ability and its comparativeness with the sample. For the proposed TDV-SIM, the SR information is extracted from raw images by the conventional frequency-extracting process and then integrated into the TDV network for artifact suppression. By combining the advantages of conventional physical model-based algorithms with DL-based algorithms, TDV-SIM outperforms existing reconstruction methods in removing artifacts associated with regions of low SNR while retaining sharp-contrast intricate structures. For example, the reconstructed actin filaments and ER of TDV-SIM have an 80.1% decrease in the background artifacts compared with Hessian-SIM and a 24.3% increase in signal fidelity compared with DFCAN. Indeed, all current DL-based reconstruction methods generate blurred mitochondrial cristae structures,24^,25 highlighting the difficulty of pure data-driven methods in predicting irregular and complicated structures in constant changes. Under such circumstances, incorporating physical constraints about the image formation process becomes critical, as we show here. Therefore, TDV-SIM has significant advantages over pure DL methods in the face of samples with intricate and dynamic structures.

However, the current TDV-SIM has limitations. On the one hand, inherited from conventional restoration algorithms, better reconstruction results depend on choosing ideal parameters. Through comparative experiments, we set the optimal hyperparameters to be 2.5 and 25 for λ and T respectively, which are applicable in most cases. However, we may need to introduce adaptive mechanisms to achieve an optimized adjustment step in the future. On the other hand, we cannot apply the current neural network regularization term to different specimens and imaging modalities. Future exploration of other regularization terms more generally applicable to different samples may further improve the adaptability and robustness of our method. Besides, TDV-SIM aims to recover the real signal from the noisy raw images. In the second-order spectrum of NL-SIM, excess noise renders signals in the reconstructed SR image to be discontinued, even with the TDV-SIM. However, it will not produces hallucinative signals such as the pure DL method.

Starting from a hybrid angle, TDV-SIM presents a novel solution for high-resolution and HiFi SR-SIM reconstruction from low SNR images. Endorsed with reduced photon dosage and associated phototoxicity, improved imaging speed, and extended imaging duration, TDV-SIM will be crucial for SR imaging subcellular structure dynamics in live cells.

Jianyong Wang received his bachelor’s degree in mechanical design, manufacturing, and automation from the University of Electronic Science and Technology of China in 2019 and his master’s degree in software engineering from the School of Software and Microelectronics of Peking University in 2022. His research interest is super-resolution structured illumination microscopy.

Junchao Fan received his bachelor’s and PhD degrees in engineering from Huazhong University of Science and Technology in 2014 and 2020, respectively. He is an associate professor at Chongqing University of Posts and Telecommunications. His research is focused on the imaging processing and reconstruction algorithm of computational imaging.

Bo Zhou received his bachelor’s degree in mechanical design, manufacturing, and automation from Central South University in 2017 and his master’s degree in software engineering from the School of Software and Microelectronics, Peking University in 2020. Currently, he is a PhD student at Cell Secretion and Metabolism Laboratory, Institute of Molecular Medicine, Peking University. His research interests are the reconstruction algorithms of super-resolution fluorescence microscopy.

Xiaoshuai Huang received his bachelor’s degree in science from Wuhan University in 2013 and his PhD from Peking University in 2018. He is an assistant professor at Peking University. From 2018 to 2020, he was trained as a postdoctoral research fellow at Peking University. His research is focused on super-resolution microscopy and cell biology.

Liangyi Chen is a Boya Professor at Peking University. He majored in biomedical engineering as an undergraduate at Xi’an JiaoTong University and a PhD student at Huazhong University of Science and Technology. His lab focused on developing state-of-the-art imaging techniques, including ultrasensitive Hessian structured illumination microscopy, super-resolution fluorescence-assisted diffraction computational tomography, sparse deconvolution enabled mathematical superresolution, and fast high-resolution miniature two-photon microscopy for brain imaging in freely behaving mice. He is also a guest professor at Université PSL and École Normale Supérieure.