Organelles are implicated in multiple cellular life activities, and their metabolism or dysfunction is closely associated with the development and metastasis of cancers.1 The explorations of subcellular structures and their abnormal states facilitate insight into multiple pathological mechanisms, which are expected to achieve the early diagnosis and effective therapy of diseases.2^,3 Hence, noninvasive high-throughput microscopic imaging is of great significance for attaining the precise detections of subcellular features and high-content quantitative analysis of multiple events in a large population of cells.4^–7

Due to the low absorption or weak scattering of almost pure phase objects, the unstained cells have difficulty generating sufficient intrinsic contrast.8^–10 Though these samples can be visualized by specific dyes or fluorescent labels, the phototoxicity and photobleaching of exogenous agents could damage the cell structure and prevent the long-term observation of live cells.11^,12 Recently, quantitative phase imaging (QPI) has gained a lot of attention because of its unique capability to quantify the phase delay of unlabeled biological specimens in a nondestructive way.13^,14 As a well-established deterministic QPI approach, the transport of intensity equation (TIE) was originally derived in the monochromatic coherent illumination situation and linearized to the phase retrieval problem under the paraxial approximation.15^,16 The phase retrieval methods based on TIE enable quantitative phase microscopy by simply measuring intensities at multiple axially displaced planes. TIE has emerged as a promising QPI tool owing to the advantages of noninterferometric imaging that it is free of phase-unwrapping, and Köhler illumination compatibility within a brightfield (BF) microscope.17^–19 However, the achievable space-bandwidth product (SBP) of TIE is fundamentally restricted by the coherent optical system, exacerbating the tradeoff between image resolution and field of view (FOV).7

Generally, to increase the imaging system throughput, a low numerical-aperture (NA) objective is used to acquire a wider FOV and a larger NA illumination is applied for extending the maximum achievable resolution beyond the coherent diffraction limit. However, the original TIE is limited to the circular BF illumination and the phase contrast progressively vanishes as the illumination NA (NAill) approaches the objective NA (NAobj).20^,21 Partial coherence expands the maximum obtainable imaging resolution but aggravates the noise-to-resolution compromise since the decrease in illumination coherence tends to weaken the phase effect.22 Due to the deviation from the coherent linearized model, the TIE method has difficulty retrieving accurate results with incoherent diffraction-limited resolution under a circle-shaped BF illumination (NAill=NAobj).15^,16 This resolution-FOV tradeoff deficiency limits the applications of TIE-based QPI methods in high-throughput biomedical fields with high-SBP needs, including cancer diagnostics and drug development.5^,7

To alleviate the above problem, researchers suggested to explicitly interpreting partial coherence by generalizing the coherent contrast transfer function to the weak object transfer function (WOTF) of a partially coherent system.20^,23 As is predicted by the WOTF analysis, the high-resolution and high-contrast QPI can be achieved via boosting the TIE phase transfer function response within the theoretical framework of a transmission cross-coefficient based on the illumination modulation.24^,25 In the annular illumination-based TIE (AI-TIE),25^,26 the image resolution can be extended to 2NA because of the robust transfer function response at high frequencies when the circular illumination aperture is replaced with an annular one (where the NAill is equal to the NAobj). Nevertheless, the SBP of AI-TIE is still restricted within the incoherent diffraction limit. To meet the needs of noninvasive high-throughput imaging, it is hoped that TIE-related methods can exceed this limit to acquire high-resolution and wide FOV quantitative phase microscopy.

To the best of our knowledge, TIE is always limited by BF illumination,15^,16^,26 and the maximum attainable imaging resolution is restrained to the incoherent diffraction limit when matched annular illumination is used.25 Fundamentally, the resolution-to-FOV tradeoff of TIE-based QPI methods originates from the imperfection of the simplified and linearized imaging model, which ignores partial coherence or introduces strict assumptions. In general, higher NAill implies a larger spectral high-frequency cutoff frequency, increasing the ability of the microscope to record high-frequency details of the specimen. Specifically, the partial coherence in darkfield (DF) imaging (NAill>NAobj) extends the maximum achievable imaging resolution beyond the incoherent diffraction limit. But the image formation model of DF imaging transforms into a nonlinear inverse problem under the nonparaxial condition, which cannot be solved by TIE because of the model mismatch. Consequently, high-throughput TIE-based QPI surpassing the incoherent diffraction-limited resolution has not been reported so far.

In this work, we propose a hybrid BF and DF transport of intensity approach, termed HBDTI, to obtain high-throughput QPI with a large SBP. HBDTI uses a low-NA objective to acquire two through-focus intensity stacks corresponding to BF and DF illuminations with a programmable light-emitting-diode (LED) array. The forward imaging model of HBDTI is established to depict the captured intensity stacks under the partially coherent illuminations for both BF and DF imaging precisely and elegantly. Based on an iterative phase retrieval algorithm with coherence model decomposition, the synthetic aperture process of HBDTI combines BF illumination with high-angle DF illumination to synthesize a high-resolution and large-FOV complex amplitude. The effectiveness of HBDTI is validated by measuring the USAF resolution target and biological samples both with absorption and phase or pure phase. It is demonstrated that HBDTI achieves a ∼7.19mm2 wide FOV of 488 nm half-width resolution that is ∼2.5× higher than the incoherent diffraction limit, corresponding to ∼30.2megapixels SBP. Experimental results show the capability of HBDTI for high-SBP QPI to attain accurate detections of subcellular structures from large groups of cells across multiscales.

The essential idea of computational phase microscopy based on TIE is to build an accurate forward image formation model and then implement the phase retrieval algorithm via solving the corresponding inverse problem.19^,27 However, for partially coherent illumination, the imaging system is nonlinear in either amplitude or intensity, making phase retrieval complicated.25 To simplify the image formation process, weakening the effect of partial coherence and implementing restrictive assumptions, such as the paraxial approximation and the weak defocusing, are often applied to linearize the TIE-related phase retrieval problems. In linearized imaging models, quantitative phase distributions can be directly reconstructed by Fourier space deconvolution in one step by simplifying the model complexity to decouple the phase solution from the model explicitly.

For diffraction-limited optical imaging systems, higher NAobj and NAill raise the available angles and expand the accessible object frequency, thereby having a more powerful capability to resolve fine specimen details. However, when NAill>NAobj, the image formation process, from linear BF imaging (paraxial) to nonlinear DF imaging (nonparaxial), cannot be solved by the TIE linearized model.28^,29 To obtain the imaging resolution over the incoherent diffraction limit, it is necessary to break the strict approximations of the original TIE and establish a more realistic forward physical model. The iterative solutions, such as Fourier ptychographic microscopy (FPM), can bypass the analytical modeling of the complex inverse problems and achieve high-resolution complex amplitude recovery by performing forward imaging models only. Thus, we establish an HBDTI forward imaging model to describe the measured intensity for both BF and DF imaging in the partially coherent fields precisely in an elegant way. HBDTI doesn’t require the restrictive assumptions like TIE as it considers the partial coherence of the image formation and utilizes coherence model decomposition in an iterative process.

In the actual measurement of the intensity image, the dynamic range of BF and DF intensity images differs by several orders of magnitude, and the signal-to-noise ratio (SNR) between corresponding intensity images is quite different as well.30^,31 Thus, the intensity images are separately captured under these two illuminations for better SNR and the capture process takes full advantage of the detector dynamic range, as illustrated in Fig. 1(b). It is worth mentioning that acquiring three sets (or more) of data according to the illumination NA can further effectively enhance the imaging SNR. But considering that the difference in the dynamic range of DF images with different NAs is much smaller than that in the BF case, the improved SNR performance of this strategy is relatively limited and increases the complexity of data acquisition. Hence, the proposed HBDTI captures only two sets of through-focus intensity stacks in BF and DF for the tradeoff of SNR and acquisition efficiency. The utilization of through-focus in TIE-based QPI is a traditional approach to introducing the imaginary component of the optical transfer function for improving phase effect, thereby the phase information can be transferred into the defocused intensity images.23^,32 Based on the above considerations, we capture two through-focus intensity stacks under BF and DF illuminations as the inputs of the convergent iterative process for solving the nonlinear image formation of HBDTI.

To illustrate the advantages of HBDTI from the perspective of optical principle, we compared the optical system of the traditional FPM method and that of the proposed HBDTI method, as shown in Fig. 1. According to Fig. 1(a), FPM requires a variably illuminated in-focus intensity stack (∼200 frames) with both BF and DF intensity images,33^–36 which are captured in low-SNR under point illumination. In terms of the matched illumination condition,37 FPM recovers low-frequency components only when the illumination angles are matched to the cut-off angles allowed by the objective pupil function. In contrast, HBDTI captures two through-focus intensity stacks (∼100 frames) under BF and DF illuminations. Using the discrete circle and complementary-shaped patterns shown in Fig. 1(b) rather than point sources, HBDTI obtains high-SNR intensities to ensure high quality and robustness of phase retrieval. Moreover, HBDTI bypasses the strict requirement of the matched illumination condition via through-focus scanning, avoiding the missing of low-frequency components.38 By contrast, HBDTI can achieve high-throughput quantitative phase microscopy as FPM based on a relatively small amount of data without the need to consider the matched illumination condition. In particular, the advantages of HBDTI compared with different QPI techniques are shown in Sec. 2 in the Supplementary Material.

We implement the HBDTI recovery algorithm by measuring two sets of Nz through-focus intensity stacks IBFcap and IDFcap corresponding to the BF and DF illuminations [Fig. 2(a)], where z is the defocus distance and zstep is the step size of the intensity stack. As shown in step 2 in Fig. 2(b), the captured in-focus intensity is upsampled for the initialization of the object complex amplitude U. Then, a series of high-resolution defocused intensity stacks are obtained through numerical propagation under corresponding LED illumination angles [step 3 in Fig. 2(c)], and the calculated low-resolution intensities after the downsampling of pixel binning can be treated as the input of the following intensity constraints process. In step 4 [Fig. 2(d)], the intensity images under different illumination angles are summed up to form the two intensity stacks IBFcal and IDFcal under corresponding BF and DF illuminations by coherent mode decomposition.34^,39^,40 The measured intensity stacks are divided by the two stacks as the factor to update the complex amplitude corresponding to various defocus distances. Inspired by the strategy of FPM,33 we apply the schemes of synthetic aperture and multiplexing to the difference map [step 5 in Fig. 2(e)],41^–44 and the intensity iterative constraint process is compatible with high-angle DF illuminations, which significantly expands the accessible object frequency. The complex amplitude stacks at different planes are backpropagated to the in-focal object plane and then these propagated complex amplitude stacks are synthesized in the Fourier domain to update the object complex amplitude. Finally, the high-resolution object complex amplitude can be retrieved by repeating step 3 to step 5 until the intensity constraint is satisfied upon convergence. More details are provided in Sec. 1 in the Supplementary Material.

It is worth mentioning that the HBDTI method proposed in this paper is different from traditional FPM.33^,35^,36 In FPM, it requires a variable-illuminated intensity stack (∼200 frames) including DF features at the in-focus plane under point illumination (leading to low-SNR intensity).33^,45 Incorporating ptychographic phase retrieval and coherent synthetic aperture, FPM allows high-SBP imaging beyond the incoherent diffraction limit. But in terms of the matched illumination condition,37 FPM recovers low-frequency components only if the illumination angles match the cutoff angle allowed by the objective pupil function. Differently, HBDTI captures two through-focus intensity stacks (∼100 frames) under BF and DF illumination with discrete circular and complementary-shaped patterns for high-SNR intensity. By merging synthetic aperture and multiplexed illumination,46 the achievable resolution of HBDTI can exceed the incoherent diffraction limit for high-SBP imaging. In contrast to FPM, as a propagation-based phase recovery method, the low-frequency phase components of HBDTI can be completely transferred by z-scanning. Accordingly, HBDTI bypasses the strict requirement of the matched illumination condition by through-focus scanning, avoiding the loss of low-frequency components.38 Therefore, HBDTI can achieve the same high-SBP reconstruction capability as FPM only based on simpler operation and a smaller amount of data without the need for the matched illumination condition.

We perform the simulation under the 632 nm wavelength LED illumination with a 5.5μm pixel size detector and the 4×, 0.1 NA objective to verify the feasibility of the proposed HBDTI approach. The simulated 9×9 LED array of 120 mm below the sample is shown in Fig. 3(b), with a 6.5 mm distance between adjacent units. The original low-resolution data were defined with 64×64pixels resolution, and the recovered 192×192pixels high-resolution intensity IHR was calculated in iteration number Niter=10. As illustrated in Fig. 3(c), the details in the simulated low-resolution intensity image were severely blurred due to pixel binning. The HBDTI retrieval result in Fig. 3(d) presents more features than that in Fig. 3(c), such as the folds of clothes and the sideburns of the cameraman.

Moreover, the high-throughput microscopy capability of HBDTI was validated by measuring the USAF resolution target, blood smear, and unlabeled Henrietta Lacks (HeLa) cells (both quantitative absorption and phase distributions) displayed in Figs. 4–6. The experiments were implemented on a commercial microscope (IX83, Olympus, Japan) that was equipped with a 4×, 0.16 NA objective (PLN, Olympus, Japan) utilizing a motorized focus drive with a minimum step size of 10 nm, an industrial camera (The Imaging Source DMK33UX183, pixel resolution 5472×3648, pixel pitch 2.4μm) and a programmable LED array. Each LED (RS-1515MBAM, Nationstar) approximately consumes 40 mW of power and provides spatially coherent quasimonochromatic illumination through three individual channels (central wavelength red 632 nm, green 504 nm, blue 460 nm, and ∼20nm bandwidth). The LED array is driven by a self-developed circuit with a field programmable gate array (FPGA) unit (EP4CE10E22C8N, Intel FPGA, United States) to provide logical control. The central 9×9 LED array with a distance of 2 mm between adjacent units was placed 12.3 mm above the sample, offering the largest effective NAill≈0.6 (in the case of BF illumination, NAill≈0.16). In the experiments, in total, 100 intensity frames are acquired within a 3D intensity set (5472×3648×100) for 50 axial z-slices under BF and DF illuminations achieved by a 9×9 LED array within 5.5 s acquisition time (55 ms exposure time for each frame under 4× objective lens). To improve computational efficiency, we adopt a nonmechanical image stitching algorithm of image segment recombination to reduce computational cost. Thus, in a normal laptop without GPU acceleration, our proposed method takes ∼1min at ten iterations for 100 intensity images in the experiments.

To demonstrate the throughput of HBDTI, the 1951 USAF resolution target (Ready Optics Company, United States) was captured with a 4×, 0.16 NA objective under illumination λ=632nm as illustrated in Fig. 4. According to Sec. 3 in the Supplementary Material, the two sets of Nz=50z-axis intensity stacks were measured in zstep=5μm under BF and DF illuminations. For the measured in-focus intensity shown in Fig. 4(a2), the highest distinguishable resolution target bars were element 5 in group 8 [Fig. 4(b1), 1.23μm half-pitch resolution]. Compared with the measured BF intensity and the quantitative profile in Figs. 4(b1) and 4(c1), all the unit details of group 8 were resolved in the HBDTI recovery result in higher contrast shown in Figs. 4(b2) and 4(c2). As depicted in Fig. 4(b2), the highest resolvable target bars of HBDTI retrieval result were element 1 in group 10, with a half-pitch resolution of 0.488μm. The maximum resolution of the experimental result presented the success of HBDTI to retrieve high-frequency features consistent with the theoretical half-pitch resolution of ∼0.452μm, λ/(NAill+NAobj).47 It was validated that HBDTI achieved high-resolution imaging of ∼2.5× higher than the incoherent diffraction limit in the 0.16 NA objective, and the corresponding SBP was up to 30.2 megapixels with ∼6.25× improvement under the 4× FOV of ∼7.19mm2.

HBDTI quantitatively recovers both phase and amplitude easily using two sets of defocused intensity stacks, which was confirmed by measuring the blood smear (Carolina Biological Supply Company, Burlington, North Carolina, United States) based on the 4×, 0.16 NA objective (see Fig. 5 and Video 1). The low-resolution BF in-focus intensity shown in Fig. 5(a) was combined by the intensity images captured under quasimonochromatic illumination at λ=632nm (red), 504 nm (green), and 460 nm (blue), respectively. Applying the same experimental and computational parameters as the first experiment, the blurred edges and missing details of the blood cells in Fig. 5(b1) were retrieved using the proposed HBDTI method [Fig. 5(b3)]. Compared with the quantitative distribution in the measured intensity [Fig. 5(e1)], the profile of the blood cell was resolved clearly in the HBDTI recovery result both in intensity and phase [Figs. 5(e2) and 5(e3)]. In addition, as shown in Figs. 5(c1)–5(d2), the sharp red blood cells boundaries and the clear white blood cells’ internal particles were distinguished under a large FOV containing ∼80,000 cells [Fig. 5(a)],48 proving the capability of HBDTI for precise detections of subcellular structures across the submicron scale to millimeter scale.

Our technique visualizes and provides high-resolution phase delay quantification of the unstained HeLa cells as the phase object, which was proved under the 4×, 0.16 NA objective with illumination λ=632nm (see Fig. 6 and Video 2). HeLa cells were seeded (at an initial density of 300cells/cm2) in a 35 mm glass-bottom Petri dish (No.0 Uncoated Coverslip, 10 mm Glass Diameter, MatTek Corporation, P35G010C) using DMEM high glucose with pyruvate (4.5 g 1:1 glucose, with GlutaMAX supplement, Gibco, Thermo Fisher Scientific or Roti-CELL DMEM, Roth) supplemented with 10% fetal bovine serum and 1× penicillin-streptomycin (both Gibco, Thermo Fisher Scientific). Cells were incubated at 37°C in a humidified atmosphere of 5% carbon dioxide for 8 h to allow attachment. After that, cells were fixed in phosphate-buffered saline (PBS) buffer (nm=1.34) on a microscope slide to preserve the cell morphology. According to Sec. 3 in the Supplementary Material, the two sets of Nz=50z-axis intensity stacks were also measured in zstep=5μm under BF and DF illuminations. We adopted a nonmechanical image stitching algorithm of image segment recombination to reduce computational costs.33 As illustrated in Fig. 6(a), about 4000 HeLa cells with subcellular details were observed in the HBDTI recovery phase under a large FOV of ∼7.19mm2. For better visual effects, we normalized the dynamic display range of the BF intensity and DF intensity, respectively. As shown in Figs. 6(b1) and 6(b2) and Figs. 6(c1) and 6(c2), it was confirmed that the intensity images acquired by HBDTI based on the taking full advantage of the detector dynamic range have a high SNR. As shown in Figs. 6(b3) and 6(c3), because of the low spatial coherence of circle-shaped BF illumination, TIE retrieved blurred cell boundaries and subcellular details with low contrast. Meanwhile, the missing details in the TIE retrieval results were accurately recovered by utilizing the proposed HBDTI approach in a wide FOV [Figs. 6(b4) and 6(c4)], achieving noninvasive high-throughput imaging for the high-content analysis of subcellular structure detections.

We have proposed a hybrid BF and DF transport of intensity approach for high-throughput QPI called HBDTI to solve the dilemma between the resolution and FOV. HBDTI extends the achievable imaging resolution beyond the incoherent diffraction limit via solving the nonlinear phase recovery problem under DF illumination, which is difficult for the linear formula of TIE. Utilizing two through-focus intensity stacks under BF and DF illuminations as input, HBDTI solves the nonlinear DF image formation model based on the coherent mode decomposition, which provides high-resolution recovery exceeding ∼2.5× the incoherent diffraction limit in a large FOV. The successful performance of HBDTI has been demonstrated by the recovery of the USAF absorption target and biological specimens both with absorption and phase or pure phase. The experimental results have verified that the proposed HBDTI can achieve a 7.19mm2 wide FOV with a half-width resolution of 488 nm, achieving an improvement of 6.25× in SBP from ∼5megapixels to ∼30.2megapixels. In the traditional TIE-based QPI method, whose illumination is limited in BF, its maximum attainable imaging resolution is at most 2NA. By combining BF with DF illuminations, the proposed HBDTI method has the capability to achieve the maximum resolution of about 5NA with a 5× increase in the coherent diffraction limit. HBDTI shows the high-throughput ability to record large FOV images without degradation of the spatial resolution and has the potential in delineating subcellular structures in large-scale cell studies applicable for relatively thin objects. Further efforts, such as the differential collection of BF and DF images to relieve the data requirement and reduce the capturing time, are needed to promote the high-speed implementation of HBDTI in large-group live cell analysis.

Linpeng Lu received her BE degree from Nanjing University of Science and Technology. She is a fourth-year PhD student at Nanjing University of Science and Technology. Her research interests include computational microscopy, phase retrieval, and quantitative phase imaging. She is a member of SPIE and Optica.

Jiaji Li is a postdoc at the School of Electronic and Optical Engineering, Nanjing University of Science and Technology. He received his PhD from Nanjing University of Science and Technology in 2020. He was a visiting PhD student at the Department of Electrical and Computer Engineering, Boston University, from 2018 to 2019. He is currently a member of the Smart Computational Imaging Laboratory. His research interests include phase imaging and diffraction tomographic imaging. He is a member of SPIE and Optica.

Yefeng Shu is pursuing his PhD at Nanjing University of Science and Technology. His research interests include super-resolution microscopy, phase retrieval, and computational imaging. He is a member of SPIE and Optica.

Jiasong Sun received his BS degree from Soochow University in 2012 and his PhD from Nanjing University of Science and Technology in 2019. He is now an associate professor at the Department of Electronic and Optical Engineering. His research interest focuses on computational imaging, phase retrieval, super-resolution microscopy, diffraction tomography, and optical imaging. He is a member of SPIE and Optica.

Jie Zhou received his BE degree and is pursuing his ME degree at Nanjing University of Science and Technology. His research interests include deep learning-based phase retrieval and super-resolution microscopy. He is a member of SPIE and Optica.

Edmund Y. Lam received his BS degree, his MS degree, and his PhD, all in electrical engineering from Stanford University, in 1995, 1996, and 2000, respectively. He was the 49th PhD graduate of Prof. Joseph W. Goodman. He is now a professor in electrical and electronic engineering, the Director of the Computer Engineering Program, and the Founding Director of the Imaging Systems Laboratory at the University of Hong Kong. A recipient of the IBM Faculty Award, he is also a fellow of Optica, SPIE, IEEE, IS&T, as well as the Hong Kong Institution of Engineers.

Qian Chen received his BS, MS, and PhD degrees from Nanjing University of Science and Technology. He is currently a professor and vice-principal at Nanjing University of Science and Technology. He has been selected as Changjiang Scholar Distinguished Professor. With broad research interests in photoelectric imaging and information processing, he has authored more than 200 journal papers. His research team develops novel technologies and systems for non-interferometric quantitative phase imaging and high-speed 3D sensing and imaging, with particular applications in national defense, industry, and bio-medicine. He is a member of SPIE and Optica.

Chao Zuo received his BE and PhD degrees from Nanjing University of Science and Technology (NJUST) in 2009 and 2014, respectively. He was a research assistant at the Centre for Optics and Lasers Engineering, Nanyang Technological University, from 2012 to 2013. He is now a professor at the Department of Electronic and Optical Engineering and principal investigator of the Smart Computational Imaging Laboratory, NJUST. He has broad research interests in computational imaging and high-speed 3D sensing and has authored over 160 peer-reviewed journal publications. He has been selected for the Natural Science Foundation of China for Excellent Young Scholars and the Outstanding Youth Foundation of Jiangsu Province, China. He is a senior member of SPIE and Optica.