1Aerospace Laser Technology and System Department, Shanghai Institute of Optics and Fine Mechanics, Chinese Academy of Sciences, Shanghai 201800, China
2Wangzhijiang Innovation Center for Laser, Aerospace Laser Technology and System Department, Shanghai Institute of Optics and Fine Mechanics, Chinese Academy of Sciences, Shanghai 201800, China
3School of Physics and Optoelectronic Engineering, Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou 310024, China
4School of Data Science, Fudan University, Shanghai 200433, China
5Center of Materials Science and Optoelectronics Engineering, University of Chinese Academy of Sciences, Beijing 100049, China
【AIGC One Sentence Reading】:We propose a single-shot SR imaging method using a GISC camera, enhancing resolution beyond Rayleigh's criterion in H-D light-field space.
【AIGC Short Abstract】:This study introduces a single-shot super-resolution imaging method using a ghost imaging camera with sparsity constraints, encoding high-dimensional light-field information into a 2D speckle pattern. It enhances spatial resolution beyond Rayleigh's criterion and quantifies the impacts of sampling, signal-to-noise ratio, and object sparsity.
Note: This section is automatically generated by AI . The website and platform operators shall not be liable for any commercial or legal consequences arising from your use of AI generated content on this website. Please be aware of this.
Abstract
Super-resolution (SR) imaging has been widely used in several fields like remote sensing and microscopy. However, it is challenging for existing SR approaches to capture SR images in a single shot, especially in dynamic imaging scenarios. In this study, we present a single-shot SR imaging scheme that leverages discernibility in the high-dimensional (H-D) light-field space based on a ghost imaging camera via sparsity constraints (GISC camera), which is capable of encoding H-D imaging information into a two-dimensional speckle pattern detected in a single shot. We demonstrate both theoretically and experimentally that while the resolution in the H-D light-field space, characterized by the second-order light-field correlation, remains limited by light-field diffraction, the single-shot spatial resolution is greatly improved beyond classical Rayleigh’s criterion by utilizing the discernibility in the H-D light-field space. We further quantify the effects of the sampling number, signal-to-noise ratio, and object sparsity on the resolution. Our results offer significant potential for the SR observation of high-speed dynamic processes.
1. INTRODUCTION
Due to the diffraction effect of light-field propagation, the resolution of imaging systems is limited. For over a century, Rayleigh’s criterion has been the most influential principle of the resolution limit for incoherent optical imaging systems [1]. Nonetheless, various technical methods to break Rayleigh’s criterion have been proposed [2]. In structured illumination microscopy (SIM) [3] and Fourier ptychographic microscopy [4], by utilizing multiple structured illuminating light fields, the diffraction limit imposed by the finite aperture of an imaging system can be broken through post-synthetic processing. By reducing the scanning spot size of illuminating or emitting light using the near-field or stimulated emission depletion effect, point-scanning super-resolution (SR) imaging methods, such as near-field scanning optical microscopy (NSOM) [5] and stimulated emission depletion microscopy (STED) [6], successfully surpass the Rayleigh diffraction limit. Another category of far-field SR imaging techniques utilizes the single-molecule localization, including stochastic optical reconstruction microscopy (STORM) [7] and photoactivated localization microscopy (PALM) [8]. These methods estimate the positions of individual fluorescent molecules from diffraction-limited image sequences and synthesize an SR image from thousands to tens of thousands of localization images. More recently, overcoming the “Rayleigh’s curse” is investigated by utilizing quantum properties of classical and quantum light fields [9,10]. In parallel with the above physics-based SR techniques, numerous SR algorithms [11] have been developed to achieve SR images from low-resolution images, such as spectrum extrapolation [12], multi-frame SR [13], SR image using deep learning [14], sparsity prior [15,16], and wavelength multiplexing [17,18].
Traditionally, almost all SR imaging systems have been designed for the direct point-by-point imaging mode, in which the emitted light from each two-dimensional (2-D) spatial point at the object plane is directly recorded on the corresponding pixel of the detector at the image plane. The SR capability of this imaging mode lies in the ability to enhance the precision of separating two adjacent points in the 2-D space. Essentially, light fields can be fully characterized by a multidimensional plenoptic function involving the temporal, spatial, spectral, and polarization dimensions [19]. Shannon’s channel capacity of an imaging system can be described as , which is determined by the detection signal-to-noise ratio (SNR) using the Gaussian noise model and multiple optical degrees of freedom (DOFs), where , with , and denoting the temporal, spatial, spectral, and polarization DOFs, respectively [20,21]. However, the existing direct point-by-point imaging mode poses challenges when capturing image information in the high-dimensional (H-D) light-field space using a 2-D photon detector, and it will “degenerate” the DOFs of light fields in single-shot detection. Therefore, the utilization of H-D imaging channel capacity is limited, which makes it hard to achieve SR with a single-shot measurement even for existing state-of-the-art SR imaging techniques.
Different from the direct point-by-point imaging mode, ghost imaging (GI) acquires object’s image information through a “global random” encoding mode [22–25]. Specifically, GI utilizes fluctuating light fields to modulate the image information of an object, which is encoded into detectable lower-dimensional signals. Subsequently, the H-D light-field information is embedded in the high-order light-field correlation. To achieve the correlation, typical GI systems require multiple samplings in the temporal dimension [24]. In contrast, by utilizing the ergodicity of thermal light fields, the GI camera via sparsity constraints (GISC camera) [26] has been proposed to extract image information from light-field correlation in the spatial dimension rather than the temporal dimension, thereby enabling data acquisition in a single shot. So far, the GISC camera has been demonstrated to be able to simultaneously acquire objects’ information of spatial, spectral, and polarization dimensions in a single shot [27]. This implies that the GISC camera can transmit H-D image information and efficiently utilize the imaging channel capacity compared to conventional imaging systems based on the direct “point-by-point” imaging mode. Consequently, the GISC camera will provide new possibilities to break Rayleigh’s criterion using the H-D image information of objects.
Sign up for Photonics Research TOC. Get the latest issue of Photonics Research delivered right to you!Sign up now
In this study, we propose a single-shot SR imaging scheme by leveraging the object’s discernibility in the H-D light-field information acquired by the GISC camera. Through theoretical and experimental studies, our results demonstrate that while the resolution in the H-D light-field space, characterized by the second-order correlation of light fields, remains limited by diffraction, the spatial resolution can be significantly improved by leveraging the discernibility in other dimensions, such as the spectral and polarization dimensions. We theoretically analyze the overall limitations of imaging resolution in the H-D light-field space and establish quantitative relationships among the imaging resolution, SNR, number of samplings, and object sparsity level. Moreover, the relationship between the spatial SR capability of the GISC camera and the discernibility in the H-D light-field space is experimentally investigated. Our experimental results demonstrate that the single-shot SR imaging scheme can achieve 5-fold spatial resolution improvement over the Rayleigh diffraction limit. It is worth mentioning that unlike those SR imaging approaches that sacrifice resolution in other dimensions to enhance spatial resolution [28], our proposed scheme efficiently exploits the channel capacity of imaging systems, enabling SR imaging in a single shot.
2. METHOD
A. Principle of the SR Imaging Scheme Based on the GISC Camera
The SR scheme via discernibility in the H-D light-field space based on the GISC camera is shown as Fig. 1. Figure 1(a) shows the principle of the proposed SR imaging. Considering that the object consisting of two sources with H-D light-field information is imaged through a lens system, the two sources become diffraction-limited and cannot be resolved through the direct 2D point-by-point imaging mode due to the large size of the diffuse spots. In contrast, in the GISC camera, the object’s H-D information is encoded into a detectable 2D speckle pattern, which can be detected by an array detector in a single shot. By utilizing prior constraints, the H-D information with high resolution can be retrieved from the single-shot detection speckle image using the pre-calibrated system responses of the GISC camera when the system is set up. Furthermore, the SR image is achieved by synthesizing these H-D images.
Figure 1.Schematic of the proposed super-resolution (SR) scheme through the high-dimensional (H-D) light-field information based on the GISC camera. (a) Principle illustration of single-shot SR imaging. When imaged through a lens system, the object consisting of two sources with H-D light-field information becomes diffraction-limited and cannot be resolved via direct 2D point-by-point imaging mode due to the diffuse spots. In contrast, the GISC camera can capture and reconstruct the H-D light-field information from a single-shot measurement. The spatial resolution can be improved by utilizing the discernibility in the H-D light-field space, resulting in an SR image through a synthesis process. (b) Schematic of the GISC camera for SR imaging. The GISC camera consists of the conventional imaging system including lens 1, iris, and lens 2 to project the object’s information on its focal plane, the spatial random phase modulator (SRPM) to modulate the H-D image information into a detectable 2D speckle pattern, and an array detector to record the pattern record in a single shot. The proposed SR scheme consists of three processes: calibration, detection, and SR. During the calibration process, the system response of the GISC camera will be measured by scanning a point calibration source in H-D light-field space, which is done after the system is setup. At the detection stage, the H-D information of the object is detected in single-shot detection. In the SR step, the H-D image information of the object is reconstructed with the input of single-shot measurement and the calibrated system response, and further the SR image is achieved by a synthesis process.
Figure 1(b) shows the schematic diagram of the GISC camera for capturing H-D light-field information for SR imaging. Specifically, the proposed SR scheme based on the GISC camera involves three stages: calibration, detection, and SR. In the calibration stage, the system responses of the GISC camera are measured by scanning a point calibration source in the H-D light-field space. This process is performed only once when the GISC camera is set up. At the detection stage, the object with H-D information is modulated into a 2-D speckle pattern, which is captured in a single shot. Finally, the H-D image information of the object is reconstructed using the single-shot measurement and the pre-calibrated responses via an optimization algorithm, and the SR image is then achieved by a synthesis process.
As the core system of the proposed scheme, the apparatus of the GISC camera system is as follows. A conventional imaging system consisting of lens 1 with focal length , an iris with aperture , and lens 2 with focal length is used to project the object onto its focal plane. A spatial random phase modulator (SRPM) placed away from the focal plane of lens 2 is exploited to modulate the light fields of the captured H-D information, and finally the detector away from the SRPM records the modulated light signal. The SRPM plays a critical role by modulating thermal light into a spatially fluctuating pseudo-thermal light and acting as a random grating that generates uncorrelated speckles for different wavelengths. It facilitates the modulation of object’s H-D information into a detectable 2D plane, allowing the GISC camera to achieve H-D imaging in a single exposure. Basically, the H-D imaging information is contained in the second-order light-field fluctuation correlation [22–25] as where is the recorded signal of the H-D (spatial- and spectral-dimensional) image information in the detection process, represents the system response of the GISC camera measured during the calibration process, indicates that the ensemble average is calculated over the spatial dimension (this holds due to the ergodicity of thermal light fields), and * represents the convolution operation. This implies that the normalized second-order correlation function generally characterizes the capability of the GISC camera for acquiring H-D imaging information, which is further confirmed in Section 2.B. See Appendix B for a detailed derivation of Eq. (1).
After the signal detection, the H-D imaging information is reconstructed using the compressed-sensing (CS) algorithm [29,30]. In particular, the detection process can be discretized into the matrix form as where represents the sampling signals composed of the speckle intensity , the sampling matrix has columns consisting of the speckle patterns , is the sampling number, and denotes the image of the object in the spatial and spectral dimensions. The can be recovered by solving an inverse problem: where contains identical entries equal to the mean value of , consists of columns, each of which has identical entries equal to the mean value of the corresponding column of , and denotes the regularization on . To solve the ill-posed inverse problem Eq. (3), there are roughly three well-known categories of model-based algorithms: (1) optimization algorithms, (2) greedy algorithms, and (3) thresholding-based algorithms. In previous works, total variance regularization and rank minimization have been utilized for GISC spectral imaging in continuous scenes [26,27], since these priors represent a smooth property in both spatial and spectral dimensions. In contrast, our current study focuses on the super-resolution ability for sparse scenes in the H-D light-field space. Hence, we use a sparsity prior of the object in both spatial and spectral dimensions to solve Eq. (3). Specifically, we adopt -regularization for image restoration using the gradient projection for sparse reconstruction (GPSR) algorithm [31], as it offers superior noise reduction and stable reconstruction performance compared to -regularization.
After retrieving high-quality H-D image information, the SR image is finally achieved by a synthesis process, where the multiple images representing H-D information are merged and the high-resolution image is obtained. Along with this scheme, we further investigate a theory to illustrate that the spatial resolution of the GISC camera can be enhanced by utilizing the discernibility of objects in the H-D light-field space, such as the spectral space.
B. Theoretical Resolution Bounds of the Proposed SR Scheme
From Eq. (1), it can be found that actually characterizes the resolving ability within the framework of the second-order correlation ensemble average, where an infinite number of samplings are assumed. However, in practical imaging, the samplings are always finite, and the relation shown in Eq. (1) will be inaccurate. Besides, algorithms other than light-field correlation, such as CS algorithms that further exploit various kinds of sparsity prior, have been applied for H-D information retrieval in the GISC camera. Therefore, we conduct statistical resolution and algorithmic resolution analyses for the GISC camera rather than the description with .
Statistical resolution analysis: The theoretical analysis is conducted first from the perspective of statistical resolution [32,33]. Considering that the GISC camera does not satisfy the spatial invariance of point spread functions (PSFs), performing a statistical resolution study for the case of imaging only two-point sources (like what statistical resolution analyses on conventional imaging systems with spatial-invariant PSFs commonly do) is insufficient. Thus, we propose to consider imaging monochromatic point sources located at different positions in the H-D space with illumination and focus on the resolving condition between two of them, treating the response signals associated with other point sources as disturbances. We derive the spatial statistic resolution limit of the GISC camera by pursuing the Cramér–Rao lower bound (CRLB) of estimating the spatial distance . Given the detection model of the GISC camera represented by the probability density function , where consists of the detection signals in multiple pixels, is the parameter to be estimated, is used to represent the discernibility in the spectral information and supposed to be known, and the Fisher information matrix (FIM) and CRLB for estimating parameters can be calculated via and , respectively. By performing CRLB transformation and combining it with the statistical resolution principle , we achieve the expression for spatial statistical resolution of the GISC camera as where is the sampling number, DSNR denotes the detection SNR, which is defined as , , and is the measure of discernibility of the spectral space. The above expression highlights that the statistical limit of the resolving ability of the GISC camera does relate to sampling number , DSNR, sparsity level , and discernibility in the spectral information of the object, as well as the normalized second-order correlation function in the H-D light-field space. See Appendix C for more details about the statistical resolution. It is worth noting that the resolution limit determined by the CRLB is the best-case bound independent of the specific algorithm.
Algorithmic resolution analysis: Nevertheless, during the practical imaging process, some algorithms are employed to reconstruct the image, thereby influencing the achieved resolution. Therefore, we investigate an algorithmic resolution analysis. In this analysis, the case of imaging monochromatic point sources is considered. The analysis is conducted by combining the relationship between and the mutual coherence of the system’s transmission matrix with the exact recovery condition in the noisy scenario [34] in the CS paradigm [29,30]. The derived expression for the algorithmic resolution is formulated as below: as long as the H-D distance between each pair of point sources among those points in the H-D space satisfies the information of those point sources can be well retrieved and, thus, can be resolved. Please refer to Appendix D for more details. Similar to the statistical resolution, the algorithmic resolution also depends on the sampling number , DSNR, and sparsity level . Because the exact recovery condition on is stringent to ensure signal retrieval in the worst case, the algorithmic resolution given by Eq. (5) describes the worst-case resolution, which hardly happens in practical experiments.
Though the expressions for both statistical and algorithmic resolutions have been obtained, due to the complexity of both the expressions and the involved Bessel function in , a direct analytic expression for the imaging resolution similar to the Rayleigh criterion seems hard to give. Nevertheless, it can be conveniently obtained by a numerical solution once the parameters are given. To show that, the two imaging resolution bounds are visually illustrated by plotting them under different conditions including the sparsity level, sampling number, and DSNR in Fig. 2. As a reference, the Rayleigh diffraction resolution is plotted in Fig. 2(b) with a dark-red color. It clearly shows that the algorithmic resolution bounds are uniformly worse than the corresponding statistical resolution bounds, while both bounds are superior to the Rayleigh resolution. Also, both spatial resolution bounds improve as the discernibility in the spectral space increases, demonstrating the feasibility of the GISC camera in improving the spatial resolution by utilizing the discernibility in the H-D light-field space. Moreover, both resolution bounds get better when the sampling number or DSNR gets larger, and they get worse as the sparsity increases, indicating that the super-resolution ability is related to the object’s sparsity. Especially, when the object is not sparse, namely the sparsity of the object is very large, both resolution bounds of the GISC camera without exploiting the H-D space discernibility will approach the classical Rayleigh’s limit in the noise-free case (see Appendix C.2 and Appendix D for detailed analysis). Our theoretical results also reveal that, for objects with the same wavelength, spatial resolution highly depends on the DSNR, the object’s sparsity (), and the sampling number. Both a smaller and a higher SNR can lead to an improved spatial resolution, which is consistent with previous work [35].
Figure 2.Illustration of both statistical resolution (a) and algorithmic resolution (b) in the H-D space (spatial, spectral) under different conditions including sparsity level , sampling number , and DSNR = 100 (20 dB), 1000 (30 dB). The parameters of the GISC camera are set as , , , μ, μ, , and the classical Rayleigh’s criterion for the lens system is μ.
In order to validate the analytical results, we perform simulations. In the simulation, the parameters of the GISC camera are set as , , , μ, μ, , and , and the working waveband is , so . First, the sampling matrix whose columns consist of speckle patterns is created, where speckle patterns are simulatively achieved for the μμ field of view (FoV) and wavelength ranging from 530 nm to 560 nm with interval 15 nm; see Appendix E for more details about the simulation process. Second, a resolution test target is built [Fig. 3(A)], and the spatial distances between each slit are, respectively, 6 μm,5 μm,4 μm, and 3 μm. To imitate the diffraction limit of the conventional imaging system, we convolve the image of the resolution test target with a PSF = , to get the diffraction-limited image [Fig. 3(B)], where the full width at half-maximum (FWHM) of the PSF is about 8 μm at wavelength 530 nm. The sampling signals for the resolution test target with or without spectral difference are created via Eq. (2). By solving Eq. (3) using the GPSR algorithm, the reconstructed result of the resolution test target without a spectral difference shows that the spatial distance below 5 μm is hard to resolve [Fig. 3(D)]. But when the resolution test target has a spectral difference, where the labels 1, 2, 3 on the resolution test target have different wavelengths [Fig. 3(C)], the spatial distance as low as 3 μm in the reconstructed result can be recognized [Fig. 3(E)]. Specifically, Fig. 3(F) reveals that the spatial resolution of the GISC camera is obviously improved with the spectral difference.
Figure 3.Simulation results for the resolution target under the noiseless case. (A) Resolution test target. The width of each slit is 2 μm, and the spatial distances between each slit are 6 μm, 5 μm, 4 μm, and 3 μm, respectively. The purple scale bar is 8 μm, corresponding to the FWHM of PSF of conventional imaging system in the GISC camera. (B) Diffraction-limited image recorded by the conventional imaging system. (C) Wavelength of each slit labeled 1, 2, 3 on the resolution test target (A) when the resolution test target has spectral difference. (D) Reconstructed result without spectral difference. (E) Reconstructed result with spectral difference. (F) Comparison of resolution enhancement. Intensity profiles extracted from the cross-section green lines in (B), (D), and (E).
The influence of the detection noise on improving the spatial resolution of the GISC camera is also investigated through simulations. To this end, the recovery performances of the object with and without spectral difference are compared under different noise levels. The detection noise is assumed as Gaussian white noise, and the noise level is measured by the DSNR in dB, defined as , where is the mean value of the sampling signals and is the standard variance of Gaussian white noise. In the simulation, the is set from 5 dB to 30 dB with an increasing step of 5 dB. The reconstructed images of the object with and without spectral difference through the GPSR algorithm are shown in Figs. 4(B) and 4(C). We adopt the quantitative evaluation metrics, peak signal-to-noise ratio (PSNR), and structural similarity (SSIM) [36], to evaluate the imaging quality. One can see that the spatial resolution as well as the imaging quality of the object with spectral difference is much better than that of the object without spectral difference, especially for the low DSNR, which clearly demonstrates the advantage of the object with the spectral difference in improving the spatial resolution and image quality of recovered images.
Figure 4.Simulation results in noisy environments. (A) Ground truth, diffraction-limited image, and spectral distribution of the object. (B), (C) Reconstructed images for the no-spectral difference object and spectral difference object under different detection DSNRs. (D) Recovery comparison for the no-spectral-difference object and spectral difference object through the PSNR and SSIM [36].
Figure 5 shows the imaging results of the GISC camera in improving the spatial resolution for the dot objects, which investigates the potential for fluorescent microscopy. The dot object consists of different orientation’s dots [Fig. 5(A)], and their adjacent dots emit lights of the same wavelength or lights with different wavelengths (530 nm and 545 nm, respectively). It can be observed from Fig. 5(D) that the dots with spectral difference can be resolved (see blue rectangles) and the dots without spectral difference cannot be resolved (see yellow rectangles) in the reconstructed image, which clearly demonstrates the advantage of the GISC camera in improving the spatial resolution for the application of fluorescent microscopy by exploiting the spectral difference of fluorescent emitters.
Figure 5.Simulation for dots’ object with different orientations. (A) Ground truth with two emitting wavelengths labeled with red and green colors. The nearest distance of two emitters is 3 μm. (B) Wavelength of each emitter. (C) Diffraction-limited image of the conventional imaging system. (D) Reconstructed image. The yellow rectangles and the blue rectangles denote the reconstructed images without spectral difference and with spectral difference, respectively.
The experimental setup of the GISC camera is outlined schematically in Fig. 6. Ahead of practical imaging, the impulse responses are measured through a calibration process, which involves employing artificially generated monochromatic point sources. After calibration, the imaging object is introduced, and the resultant response signal is captured by detector 1. The captured signal is used as input for subsequent image information retrieval. For comparison, the diffraction-limited image of the conventional imaging system is recorded on detector 2 at another optical path. The classical Rayleigh’s criterion of the conventional imaging systems at the object plane is about 22.2 μm at the incident wavelength 540 nm. See Appendix F for details about the parameter settings for the GISC camera.
Figure 6.Experimental setup of the GISC camera, Xe, xenon lamp; Mc, monochromator; OF, optical fiber; BS, beam splitter; SRPM, spatial random phase modulator. A multicolor object through a conventional imaging system, which consists of a lens 1 with , an iris with aperture size , and a lens 2 with , is modulated into a speckle image by the SRPM, where and . The speckle image is expanded by a relay lens to match the pixel size of detector 1. The diffraction-limited image through the conventional imaging system is recorded on detector 2 by another optical path via a BS.
Two-point super-resolution: In the first experiment, two-point sources with different spatial distances and different spectral gaps are constructed. Specifically, two-point sources with three different spatial distances 4.0 μm, 4.6 μm, 5.1 μm corresponding to spectral gaps 23 nm,17 nm, 9 nm around the central wavelength 540 nm, respectively, are constructed. As shown in Figs. 7(A) and 7(E), the classic Richardson–Lucy deconvolution method [37] cannot resolve the two-point sources through the conventional imaging system, but the GISC camera can resolve the two-point sources. Moreover, the resolved spatial distance of the GISC camera is improved when the spectral discernibility of the H-D information of the object increases. Besides the resolved spatial distance, the localization error for the point source, which is estimated by the width of the reconstructed spots, reduces as the spectral-space difference increases, as shown in Fig. 7(B). To evaluate the achieved resolution more accurately, we analyze the obtained images in Fig. 7(A) with the Fourier rolling correlation (FRC) method [38]. The corresponding FRC curves with the curve are shown in the three subplots in Fig. 7(C), respectively, from which the critical resolutions are obtained.
Figure 7.Experimental results for resolving two point-like sources. (a) Reconstruction images of critically resolved two point-like sources under three cases: ① spatial distance 5.1 μm, spectral difference 9 nm; ② spatial distance 4.6 μm, spectral difference 17 nm; ③ spatial distance 4.1 μm, spectral difference 23 nm. (b) Localization error estimation by fitting the intensity curves of pixels along the green lines in the corresponding images shown in (a). (c) The Fourier rolling correlation (FRC) method is applied to evaluate the achieved resolution; here, corresponding FRC curves with the curve are shown. (d) Comparison of the experimental resolving distance and the evaluated resolution with an error bar with two theoretical resolution bounds. (e) The diffraction-limited image from the conventional imaging system with aperture size 1.48 mm, and the deconvolution image through the classic Richardson–Lucy method.
We compare the experimental resolution with the developed resolution bounds, as shown in Fig. 7(D), where the “Experimental” relationship between the resolving distance and the spectral-space difference is from Fig. 7(A), the error bar denotes the localization error from Fig. 7(B), the “Experimental-FRC” with the orange-star marker denotes the evaluated resolution from the FRC method, and the two theoretical resolution bound curves are given by Eqs. (4) and (5), with experimental parameters as follows: the independent sampling number . It can be observed that the experimental imaging resolution is bounded by both the statistical resolution and algorithmic resolution. This is because the algorithmic resolution defined by Eq. (5) builds on the worst case in CS [34] and the statistical resolution given by Eq. (4) represents a best-case bound for resolution estimation, which is irrelevant to specific reconstruction algorithms but relies on the sampling number and DSNR. We would like to mention that the fabrication of two-point sources in our experiments results is not merely two isolated pixels in the object plane, but rather two clusters. Therefore, for the statistical resolution curve plotted in Fig. 7(D), the sparsity parameter is set as 18, where the two point-like sources are approximately two spots with pixels determined from the reference image captured by detector 2. In contrast, when considering the algorithmic resolution, which represents the minimum distance of any two points among resolvable point sources in the case of exact recovery, since in the experiment the object consists of two clusters and points inside each cluster that are not resolved, is set to 2 in this instance.
Super-resolution results of three slits: Figure 8 presents the experimental result of three slits. The spatial distance and spectral gap between each slit are 5.0 μm and 15 nm, respectively, and the spectral width of each slit is about 4.5 nm, as shown in Fig. 8(D). Figures 8(B), 8(C), and 8(E) are the corresponding diffraction-limited image of the conventional imaging system, the deconvolution image from the diffraction-limited image, and the reconstructed super-resolution image, respectively. It can be observed that, in the case of 5.0 μm spatial distance between each slit, the imaging result of the conventional imaging system is a largely diffused spot, which is limited by Rayleigh’s criterion, and the three slits cannot be resolved from that as well as its deconvolution image. In contrast, they are clearly resolved by utilizing the H-D space discernibility. It implies that exceeding four-folds over the classical Rayleigh’s criterion, is achieved via the discernibility in the H-D space in this experiment.
Figure 8.Experimental results of three slits. (A) Ground truth. The spatial distance between each slit is 5 μm; the purple scale bar is 10 μm. (B) Diffraction-limited image of the conventional imaging system. (C) Deconvolution image through the classic Richardson–Lucy method. (D) Spectral distribution of each slit. The spectral gap between each slit is 15 nm. (E) Reconstructed super-resolution image based on the GISC camera.
In a nutshell from the results, the best resolved spatial resolution of the proposed SR scheme based on the GISC camera can achieve 4 μm at the spectral gap 23 nm, which achieves 5-fold enhancement beyond the classical Rayleigh’s criterion (μ). While current imaging results still face challenges in achieving lower-bounded statistical resolution, the relationship between the resolved spatial distance and the discernibility in the space highly correlates with the statistical resolution result. Moreover, the improvement of spatial resolution through the discernibility in the spatial-polarization space has been investigated experimentally, and a similar relationship between the resolved spatial distance and the discernibility in the spatial-polarization space is presented in Appendix H.
4. DISCUSSION
Recently, GISC nanoscopy with 80 nm spatial resolution in a single frame has been demonstrated by utilizing the sparsity of fluorescence emitters [35]. In the case that conventional Rayleigh’s criterion of the microscopy lens is about 200 nm, its spatial resolution is improved over 2-fold. Different from this work, our proposed scheme utilizes the H-D information, including not only the spatial dimension but also other dimensions like spectrum and polarization, to enhance the spatial resolution. In specific, our proof-of-principle experimental results achieve an imaging resolution exceeding 5-fold improvement over the classical Rayleigh’s criterion. It demonstrates that, besides utilizing the spatial sparsity, the capability of the GISC camera to break the Rayleigh’s criterion could be further improved by capturing H-D light-field information and exploiting the discernibility in the H-D space. Also, it is worth noting that the proposed SR imaging scheme, by utilizing the discernibility of H-D light fields, is not independent of existing SR methods. Therefore, rather than serving as an alternative to existing SR techniques, it can be incorporated into other existing SR methods to assist the resolution enhancement.
Currently, there are a few single-shot H-D imaging systems. By light-field modulation, H-D light-field images can be mapped onto a 2-D detectable plane through a compressed encoding manner and retrieved through digital decoding. However, not all of them are suitable for super-resolution imaging through the proposed scheme. Specifically, currently used modulation schemes can be sorted into two categories. One is the phase modulation, like the one used in the presented GISC camera. The other is the amplitude modulation on the image plane. For example, in coded aperture snapshot spectral imaging (CASSI) [39,40], spectral images are modulated using coded mask composed binary values (0 and 1) and are reconstructed by exploiting similarities in the H-D space. Those single-shot H-D imaging systems based on amplitude modulation, however, make it hard to apply to super-resolution imaging due to the following two points. First, the amplitude modulation sampling scheme highly relies on the assumption of image continuity so that the missed H-D information during the sampling process can be filled. However, this assumption does not hold in the field of super-resolution microscopy like fluorescence microscopic imaging. Second, to guarantee a sufficient detection SNR, the amplitude-modulation-based scheme requires the spatial size of sampling units on the image plane to significantly exceed the diffraction limit of the imaging system. Namely, the spatial resolution is mainly limited by the sampling resolution, which is far from the diffraction limit, making it unsuitable for super-resolution applications.
5. CONCLUSION
In summary, we present a single-shot super-resolution imaging scheme by leveraging the discernibility in the H-D light-field information acquired by the GISC camera. We demonstrate both theoretically and experimentally that, while the “Rayleigh’s curse” induced by the diffraction effect still exists when resolving two closely located points in the H-D light-field space, it can be avoided in the spatial dimension as long as the two points are distinguishable in other dimensions of the H-D light-field space, such as the spectral and polarization dimensions. We theoretically investigate the overall limits of imaging resolution in the H-D light-field space, establishing quantitative relationships among imaging resolution, SNR, number of samplings, and object sparsity level, and quantify the spatial-resolution improvements by leveraging discernibility in the H-D light-field space. The proof-of-principle experimental results well match the theoretical resolution bounds and show that exceeding 5-fold improvement beyond the classical Rayleigh’s criterion was achieved, while the conventional deconvolution approach cannot resolve the fine structure of the object, emphasizing the necessity and effectiveness of our proposed scheme.
With the rapid development in the fields of signal processing for sparse recovery [41,42], light-field optimization based on metasurfaces [43], megapixels and polarized detection technologies [44], and the time-of-flight detection technology [45], our work offers huge potential applications for sparse signals in their native domain, such as multi-color fluorescence microscopy [46], stimulated Raman scattering microscopy [47], polarization microscopy [48], and wavelength and polarization astronomy [49]. On the other hand, for broadband spectra or continuous scenes, the idea of realizing spatial super-resolution from the discernibility in the H-D space can be generalized. For example, with the help of distance measures in the field of manifold and information geometry [50,51], it is possible to pursue a generalized representation of H-D discernibility of broadband spectra or continuous scenes to be exploited for spatial super-resolution. Our future work will be directed toward this avenue.
APPENDIX A: MATERIALS AND METHODS
Experimental setup: The GISC camera is built in an in-house optical platform, where objects are illuminated through a white light broad-band source (arc lamp 66902, Newport). Specifically, the object is imaged onto the SRPM through two commercial lenses with focus lengths 50 mm (AFT-5020MP) and 300 mm (Tamron 70–300 mm) and then modulated into a speckle image through the SRPM (Thorlabs, DGUV 10-1500). An iris is set into two commercial lenses with the aperture diameter size of 1.48 mm to control the resolution of the commercial lens. The speckle image is expanded by a relay lens (Olympus, UPlanSApo ) and recorded on the CCD camera (Andor, iKon-M). For comparison, the diffraction-limited image of commercial lens 2 is recorded on detector 2 (Allied Vision, Stingray F-504B) at another optical path split by a 10/90 (R:T) beam splitter.
There are two processes for the GISC camera to acquire the image information of the object: (1) the pre-determined calibration process and (2) the imaging process. In the pre-determined calibration process, where the sampling matrix is obtained, a monochrome point source is fabricated by using an optical fiber coupling the quasi-monochrome light from a broadband white source. The incident light wavelength varies from 540 nm to 615 nm with interval . The point source shifts at a step size μ in the μμ FoV on the object plane, and each speckle pattern is recorded to form a column of the sampling matrix . The dimensional size of the sampling matrix is , where the spatial sample number is , the spectral sample number is 11, and the sampling rate () is over 5. In the imaging process, for generating the sampling signals, the multi-color object is projected on the image plane through the conventional imaging system and then modulated into a speckle image by the SRPM. A relay lens is used to match the pixel size of detector 1 (pixel size: 13 μm) and the average speckle size of the pre-determined patterns. For comparison, the diffraction-limited image of the conventional imaging system is recorded on detector 2 at another optical path split by a 10/90 (R:T) beam splitter. The classical Rayleigh’s criterion of conventional imaging systems at the object plane is about 22.2 μm at the incident wavelength 540 nm. See Appendix F for more details about the parameter settings for the GISC camera’s setup.
Fabrication of two-point sources with different spectral intervals: The two-point sources are fabricated by two multimode bare fibers (core diameter is 62.5 μm). One side of the two multimode bare fibers is coupled with two spectral-interval quasi-monochrome light sources from the grating monochromator, which emits broadband light. By controlling the distance of the input end of the two multimode bare fibers, the two bare fibers can couple into the light source with a specific spectral interval. Their spectral distribution is detected by a spectrometer. At the outside of the two multimode bare fibers, the two fibers are mounted on two pinholes with spatial distance 200 μm. The outside of the two multimode bare fibers acts as the two-point source. We adopt an objective lens (Olympus, RMS20X) to shrink the spatial interval of the two-point sources. The different spatial intervals of the two-point sources are realized by controlling the distance between the outside of two bare fibers and the objective lens, which are qualitatively observed through reference detector 2 with big aperture size of iris (25 mm). Herein, the detection noise of detector 1 is mainly limited to the shot noise. Thus, the detection noise level can be measured by , where denotes the average photon number that can be estimated from the number of electronics recorded on the detector. Correspondingly, the experimental of the two-point object is about 24 dB.
Fabrication of three slits with different spectral intervals: In the experiment, the three-slit object with different spectral intervals is fabricated through a scanning mode. Specifically, each slit is realized by scanning a point source with the specific wavelength at a step size μ along one dimension. The spatial interval of two slits is realized to shift the point source at a step size μ along another orthogonal dimension. The spectral distribution of each slit is detected by a spectrometer, and spectral intervals of two slits are realized by coupling the different wavelength quasi-monochrome light into the fiber from a wide-band white source. For each position with a specific wavelength quasi-monochrome point source, the speckle image of the GISC camera is recorded on detector 1 in Fig. 6. The speckle image from the three-slit object with different spectral intervals is simulatively obtained by summing up the speckle images corresponding to the scanned point sources. Correspondingly, the experimental of the three-slit object is about 21 dB.
APPENDIX B: PRINCIPLE OF THE GISC CAMERA
As shown in Fig. 9, the GISC camera applies an SRPM, in conjunction with a conventional imaging system, such as the microscope and telescope. The conventional imaging system, which projects the object onto the image plane, is composed of lens 1 with focus length , lens 2 with focus length , and iris with aperture size D. The distance between lens 1 and lens 2 is L. The SRPM, which is placed after the image plane with distance , randomly modulates the light field irradiated from the object into speckle fields, and then the speckle fields are detected by a detector behind the SRPM with distance .
Considering the broadband incoherent illumination, the speckle intensity on the detection plane of the GISC camera is [26] where represents the high-dimensional () image of the object, and , which is detected in the pre-determined calibration process, denotes the system response of GISC camera corresponding to a monochrome point light source with wavelength at the position . and are the 2-D vectors on the object plane and the detection plane of the GISC camera, respectively.
From the viewpoint of GI [24], each pixel of the detector acts as a bucket detector at the test arm, for the fixed and different positions , and wavelength acts as a pixelated signal at the reference arm. The high-dimensional image of the object is recovered by calculating the second-order correlation function of the intensity fluctuations between the pre-determined reference arm and the test arm [26], where means the spatial ensemble average over the coordinate (for simplicity, the subscript is omitted in the following content). The second-order correlation function of the GISC camera can be achieved as [26] under the condition where is the center wavelength between and .
Therefore, Eq. (B2) yields where . By substituting we complete the derivation of Eq. (1) in the main manuscript.
APPENDIX C: THEORETICAL ANALYSIS ON THE STATISTICAL RESOLUTION
1. Theoretical Derivation1.A. Observation Model
Without loss of generality, the case of imaging equiluminous monochromatic point sources is considered, where is the intensity of the point sources, denotes the spatial coordinate (for convenience only one spatial dimension is considered here), and is the wave number. In particular, we focus on estimating two of those point sources while regarding response signals generated from other point sources as disturbances. From Eq. (3), the observation model of the GISC camera in this case can be written as where represent the column of matrix corresponding to the chosen point sources , respectively, denotes the disturbances brought by other points, and is the detection noise. Here, since we are pursuing the spatial resolution, is considered as the parameters that need to be estimated, while is supposed to be known and can be varied.
Assuming that detection signals on each pixel are mutually independent and the detection photon flux is relatively high, the noise can be approximated to obey a Gaussian distribution with the shot noise SNR . Besides, since the disturbance is contributed by signals from several point sources, it is assumed to be subject to a Gaussian distribution . In particular, where the circular complex Gaussian property of light fields is adopted. Also, it is worth noting that is adopted since, when considering the resolution, the differences between wavelengths are relatively small.
Thus, the likelihood function of given the detected signal is where .
Fisher Information Matrix
The CRLB for estimating can be given by calculating the Fisher information matrix (FIM),
Herein, the expectation of the second-order partial derivative with respect to can be calculated as
Considering that so that , the FIM for can be written as
By utilizing and on those non-diagonal elements, and taking the limit , the FIM can be obtained as
In statistical resolution analysis [32], estimation of the distance rather than the individual position is preferred. Hence, a parameter transform is performed here, and the FIM is accordingly transformed to be
1.C. CRLB and Statistical Resolution
Then, the CRLB for estimating can be calculated via and obtained as
For convenience of the specific calculation, is approximated as a Gaussian form [9], where . And the CRLB of estimating can be calculated as where . Finally, By using the statistical-resolution principle , the constraint expression for the spatial statistical resolution can be obtained as from which the resolution is available for specific conditions.
2. Some Analyses on the Statistical Resolution
In order to have a more intuitive perspective on the statistical resolution [Eq. (C11), let us consider it under some specific conditions. System.Xml.XmlElement
In this case, only two point sources are imaged. Equation (C11) is simplified as
Further, considering the case of , the critical spatial resolution without exploiting the H-D information can be obtained as
Considering that actually equals the mean photon number detected in each sampling pixel under the shot noise characteristics, essentially represents the total photon number received during the detection process. Thus, Eq. (C13) shows the power law relationship between the statistical resolution and the total photon number, which is consistent with the two-point statistical resolution for traditional imaging systems [32,52]. System.Xml.XmlElement
In this case, imaging non-sparse object information under the noise-free condition is considered. Equation (C11) can be simplified as where denotes the sampling ratio. If , by considering the case of , the spatial resolution limit without exploiting the H-D information can be calculated by solving the equation: obtained as which is generally consistent with the order of the Rayleigh limit. This implies that the derived statistical resolution in the H-D space can be fairly reduced to the classical 2-D spatial resolution limit under the same conditions.
3. Simulation Result of the Statistical Resolution
To validate the theoretical result of the statistical resolution under different spectral discrepancies, simulation of the detection and reconstruction of the GISC camera in imaging two points (i.e., ) is performed. Specifically, the key parameters of in the simulation are , and the conditions are . An object consisting of two points with different spectral discrepancies and different spatial distances is simulated. For each pair of spectral discrepancy and spatial distance, the two-point object is randomly generated 100 times to be detected with noise. For the image retrieval, we fully exploit the prior of two points by solving the following least-squares problem: where denote the th, th column of the sampling matrix, respectively. After the retrieval, the root mean square error (RMSE) of the reconstructed distance of two points is obtained by analyzing the corresponding 100 reconstruction results. For each spectral discrepancy, by finding the critical distance where , the simulation result of the statistical resolution is obtained.
Comparison between the theoretical result [Eq. (C11)] and the simulation is shown in Fig. 10. It can be seen that both of the two curves confirm that the spatial resolution limit of the GISC camera can be improved as the spectral discrepancy increases. Besides, the varying trends of two curves are rather consistent, which indicates that the resolution improvement from spectral discrepancy is not limitless as the spectral discrepancy increases. It is worth noting that there is a gap between the two curves, which is mainly due to the following reasons. First, there are some approximations during the theoretical derivation of the statistical resolution, e.g., the Gaussian approximation of . Second, the statistical resolution is obtained by the CRLB, which is the best-case bound independent of reconstruction algorithms, while the simulation result is influenced by the retrieval algorithm.
Figure 10.Comparison between the theoretical and simulation results on the statistical resolution under different spectral discrepancies.
APPENDIX D: ANALYSIS ON THE ALGORITHMIC RESOLUTION
Assuming that is a -sparse signal, and the nearest distance between nonzero elements of is , namely, , , where is the smallest interval corresponding to the nearest distance, and is a positive integer. Thus, the observation model Eq. (3) becomes where contains the nonzero elements of with interval in the spatial and spectral dimensions, and is a submatrix of . Then can be recovered by solving such a problem,
In CS, the reconstruction performance is related to the specific properties of the sensing matrix. One of the commonly used measures for the property is the mutual coherence of a sensing matrix defined as , where is the -norm, and denotes the th column of . And the exact recovery condition under noisy measurement [34] shows that the signal with sparsity can be exactly reconstructed from its measurement via a sensing matrix with -normalized columns, provided that the mutual coherence satisfies where is a bound for the -norm of the measurement noise (namely, ), and denotes the minimum non-zero value of .
The relationship between and the normalized second-order correlation can be established from the following equation: where denotes the practical normalized second-order coherence function obtained with a finite sampling number . It generally deviates from the theoretical , which can be written as
Then, it can be obtained that [53] by utilizing the moment theorem of the circular complex Gaussian distribution.
Based on the above relationship, Eq. (D3) can be rewritten as
Further, to give the specific expression, the specific values of and need to be clarified. In Eq. (D3), is the bound corresponding to the measurement noise using a sensing matrix with -normalized columns; while in Eq. (D2), the sensing matrix consisting of practical light-field intensity fluctuations does not necessarily satisfy this condition, so the practical detection noise level cannot be directly substituted in. However, the practical sensing matrix can be approximately transformed into a matrix with -normalized columns by dividing all entries by a constant , and this processing should lead to an equivalent problem to be solved, where the equivalent sensing matrix approximately has -normalized columns. Then, an equivalent noise level should be used to consider the bound or, more accurately, an equivalent . Similar to the statistical resolution derivation, it is assumed that the -sparse object consists of equiluminous monochromatic point sources. In this case, we have so that
Substituting the above expression into Eq. (D7), the following equation can be obtained:
Hence, the expression that imposes the constraint on the minimum distance in the spatial-spectral dimension between each pair of point sources can be achieved as where the value of (worst case for ERC) is chosen as by considering the practical fluctuation.
APPENDIX E: SYSTEM SIMULATION OF THE GISC CAMERA
The system simulation of the GISC camera is carried out based on the MATLAB program. The propagations of light fields between optical devices are simulated via the Fresnel diffraction theorem. In the simulation, both lens 1 and lens 2 in the system are modeled as thin lenses, which obey the quadratic phase distribution, , where is the focus length of the lens, and is the wave number. The SRPM is assumed as a pure random phase screen, and its transmittance function is , where the autocorrelation function of height agrees with
In the simulation, the GISC camera contains two processes: (1) the pre-determined calibration process and (2) the practical imaging process. In the pre-determined calibration process, the sampling matrix of the GISC camera is obtained, which is made up of speckle patterns. Specifically, each 2D speckle pattern is reshaped into a vector as one column of sampling matrix . Those speckle patterns are achieved in the pre-determined calibration process of the GISC camera. In particular, a monochrome point source is placed on the object plane, and the speckle pattern is generated and recorded on the detector. By shifting the point source in the field of view and varying the wavelength of the point source in the spectral range, the whole speckle patterns are obtained. In the imaging process, the sampling signals are simulated via , where the H-D image information of object is reshaped into a vector .
APPENDIX F: PARAMETER SETTING OF THE GISC CAMERA
To choose the optimal parameters of the GISC camera, such as the , and , we perform simulations. In the simulation, the parameters of the conventional imaging system contained in the GISC camera are, respectively, , , , and . We vary one of the four parameters () by fixing other parameters and achieve simulatively a series of speckle patterns according to different spatial positions through the aforementioned system simulation of the GISC camera. Figure 11 shows the normalized second-order correlation function as a function of different spatial positions in different parameter settings. The theoretical result in Fig. 11 is the Rayleigh limit, namely, the Airy function. It can be observed that as the parameters get smaller and parameters get bigger, the is much closer to the theoretical result, which well matches our theoretical analysis on the condition, .
Figure 11.Parameter () selection of the GISC camera. (A) Comparison of the correlation coefficient curves for different parameters at fixed parameters μ, , . (B) Comparison of the correlation coefficient curves for different parameters at fixed parameters μ, , . (C) Comparison of the correlation coefficient curves for different parameters at fixed parameters μ, μ, . (D) Comparison of the correlation coefficient curves for different parameters at fixed parameters μ, μ, .
APPENDIX G: INFLUENCE OF THE DIAMETER OF THE CALIBRATION POINT LIGHT SOURCE ON THE IMAGING RESOLUTION
In the part for the principle of the GISC camera and theoretical analysis of the resolution, the calibration source is assumed to be an ideal point without size. In practice, the calibration point source is realized by an optical fiber with diameter of 16 μm. Here the effect of this practical size on the resolution is briefly discussed. Based on the principle and theoretical analysis of the proposed super-resolution scheme, the potential influence on the imaging resolution comes from the deviation of the practical corresponding to a calibration source with a definite size from the ideal corresponding to an ideal point calibration source, particularly on the spatial . Thus, we have analyzed the spatial part obtained from the experimental results and plot it as the “experimental ” together with the ideal , shown in Fig. 12. It can be found that there is no significant difference between the experimental and the ideal ; hence, the resolution is basically not affected.
Figure 12.Influence of the size of the calibration point source on .
APPENDIX H: EXPERIMENTAL RESULTS OF THE GISC CAMERA IN THE SPATIAL-POLARIZATION DIMENSION
Here, we extend the improvement of spatial resolution by exploiting the discernibility of the object in the spatial-spectral () dimension to the spatial-polarization () dimension. In the experiment, two point sources with different linear polarization states are fabricated. To be specific, by shifting the fiber that acts as a point source in the pre-determined calibration process, the two point sources with spatial distances 4 μm, 6 μm, 8 μm, and 10 μm are fabricated, and the difference of the polarized angle between the two point sources is precisely controlled by rotating a polarizer after the point source, which varies from 0° to 180°. To detect the polarization information of the object, a polarizer is placed in front of the detector. By rotating the polarizer to four different detections of polarization (0°, 45°, 90°, and 135°), a polarized detector is built to record the sampling signals (, , , ). Therefore, the object with four polarized detections (, , , ) can be retrieved by independently solving such a problem, where denotes 0°, 45°, 90°, 135°. Thus, the intensity of the object is achieved by summing and . Figure 13 presents the resolved spatial distance as a function of the polarization state’s difference . One can observe that the resolved spatial distance is also improved by increasing the difference of the polarization state.
Figure 13.Experimental results for the relationship between the resolved spatial distance and discernibility in the () space. The upper one in the figure is the diffraction-limited image of the conventional imaging system, and the bottom ones are reconstructed images.
[44] T. Yamazaki, Y. Maruyama, Y. Uesaka. Four-directional pixel-wise polarization CMOS image sensor using air-gap wire grid on 2.5−μm back-illuminated pixels. 2016 IEEE International Electron Devices Meeting (IEDM), 7-8(2016).
[47] C. W. Freudiger, W. Min, B. G. Saar. Label-free biomedical imaging with high sensitivity by stimulated Raman scattering microscopy. Science, 322, 1857-1861(2008).
[53] Z. Liu, C. Hu, Z. Tong. Some research progress on the theoretical study of ghost imaging in Shanghai Institute of Optics and Fine Mechanics, Chinese Academy of Sciences. Infrared Laser Eng., 50, 20211059(2021).