Optical micro-endoscopy is widely used in various biomedical and clinical applications. As a semi-invasive imaging tool, it enables the visualization of the hard-to-reach interior of organs and viscera of the human, as well as animal, body [1–4]. It also offers a solution to studying the brain activity of freely moving animals in vivo [5–8]. To reduce the invasive damage from the probe, the probe size needs to be constrained according to different types of applications. The stringent size requirement often leads to a compromise on imaging performance and capability. For example, it is challenging to tune the focus and perform rapid 3D imaging using a compact-size endoscope probe. In recent decades, researchers have developed many techniques to realize 3D volumetric endoscopic imaging, which is of critical importance in disease diagnosis and in vivo observations.

For endoscopic systems using small gradient-index (GRIN) lenses, the focus control can be achieved by the optomechanical actuation of these lenses [9–12]. In fiber-optic endoscopic systems, the 3D volume is typically acquired by a z-scanning process, in either the distal end or the proximal end [13,14], together with scanning in the x−y directions. The multiplexing methods [14–18] have also been developed to provide simultaneous multi-spot scanning for reaching a higher 3D imaging throughput. The above-mentioned methods, however, often need bulky optomechanical components for depth information acquisition. Recently, the single-mode fiber bundle (SMFB) with ∼50,000 small cores has been utilized to develop compact 3D endoscopic systems without using bulky photoelectric components [19–21]. Compared with GRIN lenses, it could enable the visualization of fluorescence deep inside intact tissues and organs of model organisms due to its ∼50cm length. Compared with single-core fibers, it has a higher spatial bandwidth product (SBP) due to its ∼50,000 or more fiber cores. The lateral imaging resolution of existing methods, however, is limited to about 10 μm, and the sample to be observed often may need to be spatially sparse. Realizing high-performance 3D fluorescence imaging in a compact epi-illumination modality remains a challenge by utilizing this kind of fiber bundle, whose high SBP has not been fully exploited yet.

In this work, we develop a new micro-endoscope, termed light-field micro-endoscope (LFME), for realizing snapshot 3D epi-fluorescence endoscopic imaging. We combine the use of a small imaging lens pair, a compact micro-lens array (MLA), and the SMFB to build the compact LFME system. Specifically, we apply the SMFB as an imaging probe to relay light from hard-to-reach areas, since it has the advantages of a large optical bandwidth product, deep-tissue fluorescence imaging ability, and convenience in building optical systems. A 2-mm-diameter MLA is placed at the distal fiber tip to obtain the depth information of microscopic scenes. The small imaging lens pair is applied to provide three times imaging magnification of objects.

The light-field scheme has already been integrated into endoscopic systems [22–26] to realize single-shot 3D imaging. Urner et al. proposed a light-field micro-endoscopy with a GRIN lens array [22], but their system had a very short imaging probe and applied transmissive illumination, which needed further improvement for real endoscopic detection. They recently optimized their optical design and integration to realize a practical endoscope in the reflective mode [23], demonstrating good performance by imaging non-fluorescent objects with ∼100μm 3D resolution over a large volume. Liu et al. introduced an MLA into the image plane of a conventional industrial endoscope for light-field imaging [24], but they just demonstrated a transmissive imaging system with a relatively low resolution. Xue et al. [25] designed a single-shot 3D fluorescence miniature mesoscope with a compact, lightweight, and flat system, aiming at the head-mounted platform for in vivo imaging in freely moving animals. Yanny et al. [26] realized a high-speed 3D fluorescence miniature by using a GRIN lens and an optimized phase mask, realizing high-resolution and large volume imaging; but their method required a pre-calibration step and had a relatively short imaging probe.

Our developed SMFB-based light-field endoscope is simultaneously provided with a small size and long imaging probe, epi-fluorescence imaging ability, a relatively high resolution, calibration-free characteristics, and camera framerate-limited imaging speed. The diameter of the whole LFME system is restricted by the 2-mm diameter of the imaging lens pair and MLA, which thereby has relatively low invasion for potential use like in vivo observation and intraoperative pathology. Its ∼50cm length could enable the visualization of fluorescence deep inside intact tissues and organs of model organisms. Besides, unlike conventional industrial endoscopes and GRIN-lens-based systems with hard and rigid bodies, the soft and bendable fiber bundle in use provides more convenience and flexibility in real operations, making our system well suited for many biomedical and clinical applications. We validate our LFME platform by epi-fluorescence imaging of a human skin tissue section and HeLa cells through the endoscopic probe.

In this section, we discuss the optical design of the proposed LFME. For practical use, we build up an epi-fluorescence endoscopic configuration in this work, as shown in Fig. 1(a). In the illumination path, we use a 473 nm laser source with 100 mW output power (MBL-FN-473-100 mW, Changchun New Industries Optoelectronics Tech.) as the excitation light source and apply a color filter with 525 nm wavelength and 39 nm full width at half maximum (FWHM) as the emission filter (MF525-39, Thorlabs) together with a dichroic mirror (MD498, Thorlabs) for epi-fluorescence imaging. Along the imaging direction, the system successively includes a small-size imaging lens pair, a small-size MLA, an SMFB, an objective lens, and a tube lens.

The combination of SMFB, small lens pair, and MLA works as the imaging probe for direct tissue detection. The whole probe has a 2 mm diameter and is about 50 cm long. To realize three times magnification imaging, the imaging lens pair forms a 4f system, consisting of a small lens with a focal length f=3mm and 2 mm diameter (#84–124, Edmund Optics Inc.), and another small lens with a focal length f=9mm and 2 mm diameter (#83–978, Edmund Optics Inc.). The imaging lens pair has a theoretical numerical aperture (NA) of ∼0.33 and an effective NA of ∼0.30 by experimental testing. The small-size MLA (manufactured by the Institute of Optics and Electronics, Chinese Academy of Sciences) is put on the image plane of the lens pair. The specification of the MLA in use has a 24 μm pitch size and a 117 μm focal length to code the 3D information. The SMFB is used to relay the focal plane of the MLA. We put a relay imaging system (with an objective lens and a tube lens) between the sCMOS camera (pco.edge 26, 5120×5120 pixels and 2.5 μm pixel size, LUSTER LightTech) and the SMFB, which further conjugates the focal plane of the MLA to the sensor plane. Specifically, we use a 5× objective (0.14 NA, Mitutoyo MY5X-802, Thorlabs) for image acquisition. Each micro-lens covers around 7×7 effective pixels, which correspond to the angular resolution. The SMFB in use is about 48.1 cm long and has 50,000 fiber cores, 3.3 μm pitch size, 1 mm image circle diameter, and 1.2 mm coating diameter (FIGH-50-1100N, Fujikura). Details about the optical parameters setting can be found in Appendix A.

The optical design of our system has some particularities compared with the traditional epi-fluorescence light-field microscopy. We list some key points here. (1) Speckle noises. The fiber cores of SMFB in use will introduce moderate speckles when the coherent laser source is applied for epi-illumination, which will affect the imaging quality to a certain extent. Therefore, after the laser source, we place a continuously rotating diffuser (DG10-1500-MD, Thorlabs) to reduce the light coherence, driven by a motorized rotation stage (PRM1Z8, Thorlabs). For higher light efficiency, we use a condenser lens (ACL25416U-A, Thorlabs) for converging the light, and the diffuser is placed at the focal plane of the condenser. A relay lens is then applied for collimating the light. The comparison of different imaging performance with and without this rotating diffuser will be described in Section 3.D. (2) Sampling (pixelization) effect by SMFB. If we directly apply the SMFB for imaging, the spatial resolution will have an upper limit of ∼6.6μm, determined by the 3.3 μm pitch size of fiber cores (Nyquist sampling theorem). The depth of field (DoF) will have a narrow range of ∼8μm (measured by experiments), limited by the cross-talk among different fiber cores. We will talk about these limitations in Section 3.B. However, by introducing three times imaging magnification and an MLA with appropriate parameters, our LFME can bypass the pixelization effect caused by fiber cores through a resample step. Our LFME can preserve better than 6.20 μm lateral resolution within ∼24μm DoF and gain snapshot 3D imaging ability simultaneously through reconstruction, only by trading off the field of view (FoV) from 1 mm diameter to 333 μm. We will analyze the imaging performance of our method in detail in Section 3.B. (3) Illumination PSFs after MLA. We put the MLA between the SMFB and the imaging lens pair to avoid information loss caused by the spatial sampling of fiber cores. Consequently, the excitation light will transmit through the MLA before being projected on the sample [Fig. 1(a)]. Fortunately, by comparing with the reference images, we find it will hardly change the light intensity distribution on the sample, and will not affect the final imaging performance accordingly. The details are described in the last paragraph of Section 3.D. Therefore, after the resample step, the light-field image captured by our LFME is similar to the traditional light-field image.

In this work, we add a small lens pair and a small-size MLA in front of SMFB to realize the LFME system [Fig. 1(a)]. As mentioned above, by applying a resample step, the light-field image captured by our LFME is similar to the traditional light-field image. The effectiveness of the resample step is demonstrated experimentally in Appendix B. We thus can use the traditional Richardson–Lucy (RL) deconvolution [27,28] for reconstruction or apply the improved schemes [29,30] for artifact suppression. We just need to fine-tune the parameters in the algorithms according to the characteristics of the current data. Therefore, the image formation of LFME is the same as for traditional light-field microscopy [27,28].

We denote the sample space coordinate as (x1,x2,z) and the sensor space coordinate as (s1,s2). The forward image process of LFME can be modeled as a linear projection, where the system function H maps the 3D sample space into the 2D sensor space ∑x1,x2,zH(x1,x2,z,s1,s2)X(x1,x2,z)=Y(s1,s2).(1)

Here Y is the discrete image in the sensor plane and X is the 3D distribution of the sample. The weight matrix H can be sampled from the light-field PSF [27]. It records how the photons emitted from the voxel (x1,x2,z) separate and contribute to the pixel (s1,s2), and could further be simplified via the periodicity introduced by the MLA.

For image reconstruction, we first resample and realign the captured light-field image into 7×7 angular components. Then we apply the RL deconvolution [27,28] to the captured measurement, where the simulated PSFs of LFME are used to update the sample distribution X with 3–5 iterations.

We further introduce the dictionary learning procedure [29] into our proposed LFME for artifact-suppressed and contrast-enhanced reconstruction performance. As the flow chart shows in Fig. 2, the dictionary learning reconstruction process can be decomposed into three parts: calibration, reconstruction through a few runs of RL iterations, and dictionary patching. In the calibration step, the captured light-field image is still resampled and realigned into 7×7 angular components. In RL deconvolution, we still use simulated PSFs of LFME to update the sample distribution X but with fewer iterations (2–3 times). In dictionary learning, we learn a dual dictionary pair (Dl,z,Dh,z) within the light-field model, where Dl,z (low-resolution dictionary) collects the most representative elements of previous RL reconstruction and Dh,z (high-resolution dictionary) collects the corresponding high-fidelity and artifact-reduced elements of the input 3D ground truth. The element indicates the local features of an image (corners or edges). We then apply the learned dictionaries to achieve the high-quality reconstruction from raw RL reconstruction to suppress artifacts and improve the image contrast. For more details of the dictionary learning procedure for light-field reconstruction, please refer to our previous work [29].

Fluorescence endoscopy is a promising and useful technique in observing tissues, organs, and viscera of model animals or humans [1–3,14,31,32]. One application is the detection of gastrointestinal diseases, such as autofluorescence imaging of the gastrointestinal tract [1], detection of dysplasia in ulcerative colitis [2], and detection of non-visible malignant or premalignant lesions [3]. The proposed LFME is a good alternative for providing 3D fluorescence imaging.

We exhibit the fluorescence imaging of a human skin tissue section by using the LFME system in Fig. 3. The raw image in Fig. 3(a) is the light-field measurement, captured by assembling both the imaging lens pair and MLA. The raw image in Fig. 3(b) is captured only by using the imaging lens pair and serves as a reference image, which theoretically has higher spatial resolution since no MLA is applied. However, its real imaging performance is heavily contaminated by the pixelation effect of fiber cores, which is a common problem in wide-field imaging systems using SMFB. It also has a low imaging contrast with significant background fluorescence. Benefiting from our optical design of the LFME system, after performing the resample step and RL deconvolution [27,28] to the light-field measurement, our method can resolve fine details [Fig. 3(c)] similar to that of Fig. 3(b). The recovered image is with higher quality and cleaner background, although the raw light-field measurement in Fig. 3(a) is also heavily affected by the pixelation effect and noisy background. Moreover, by applying the dictionary learning method after deconvolution [29], we can further suppress artifacts and increase the image contrast, resolving more detailed structures of the skin tissue [Fig. 3(d)]. The intensity curves in Fig. 3(f), indicated by the red and blue lines in Figs. 3(c) and 3(d), can better present the artifact suppression ability of the dictionary learning method.

We also image a USAF-1951 resolution chart attached with a fluorescent board as the object for lateral resolution evaluation of our LFME system, since we could not find a proper fluorescent chart for the performance test. As shown in Fig. 3(e3), by applying dictionary learning after deconvolution, group 7 line 1 of the resolution chart can be clearly distinguished, corresponding to at least 3.91 μm lateral resolution. The resolved lateral resolution can also be verified by the blue line of the intensity curve shown in Fig. 3(g).

In this section, we use several experiments to qualitatively and quantitatively analyze the imaging performance of the proposed LFME, such as (1) lateral resolution and FoV, (2) DoF and working distance, and (3) 3D imaging ability and axial resolution. To better demonstrate the improvement of our method, we also directly apply the SMFB (without assembling the lens pair and MLA) to fluorescence imaging as a reference for imaging performance comparisons.

As shown in Figs. 3(e3) and 3(f), our method can obtain ∼3.91μm lateral resolution by imaging a fluorescence USAF-1951 resolution chart using the dictionary learning procedure. To better characterize the lateral resolution across different depths, we capture z-stack images of the resolution chart by putting it in different z positions successively. As the reconstructed results are shown in Fig. 4(c), our method can maintain ≤6.20μm lateral resolution (at least distinguish group 6 line 3) within at least 24 μm DoF. The circular FoV of our system has about 333 μm diameter, which is determined by the 1-mm-diameter fiber and 3× imaging magnification in use. In comparison, as shown in Fig. 5, the system directly applying the SMFB for imaging has the best lateral resolution of 6.96 μm (group 6 line 2) when the fiber is closely attached to the resolution chart, and the resolution drops to 7.81 μm (group 6 line 1) when the resolution chart is placed 8 μm away from the fiber. Therefore, although the light-field configuration is applied, our method still has a better lateral resolution than directly using SMFB, which is due to our optical design to bypass the pixelization effect caused by fiber cores. The only trade-off of our method is the FoV loss. The circular FoV diameter of the single-SMFB system is 1000 μm, which is three times that of our method.

As a light-field endoscope, our method can largely extend the imaging DoF. The theoretical optimal DoF of our method can be calculated by simulations, which is determined by MTFs with the largest bandwidth. As shown in Fig. 4(d), the theoretical optimal DoF has an extremely large range. However, as mentioned before, the real optimal imaging depth range (maintaining ≤6.20μm lateral resolution) is limited to about 24 μm in practice. We guess the limitation is mainly caused by the optical aberration of the small lens pair and the non-ideal PSFs of the manufactured MLA in use. Further optimization of the optical design [33], or using experimentally calibrated PSFs or adaptive PSF calibration algorithms for deconvolution [34–36], may greatly improve the real DoF, which is left for our future work. In comparison, as mentioned above, the system directly applying the SMFB for imaging has the best lateral resolution of 6.96 μm (group 6 line 2) when the fiber is closely attached to the resolution chart, and the resolution drops to 7.81 μm (group 6 line 1) when the resolution chart is placed 8 μm away from the fiber, as shown in Fig. 5. Besides, the single-SMFB system can work only by closely attaching the object to the fiber distal end, which means it has an extremely narrow working distance (<10μm), greatly limiting its applications. On the contrary, the proposed LFME can release this limitation and greatly improve the working distance to around 3 mm, which is determined by the lens pair, having a focal length of 3 mm on the side close to the sample.

Here we test the 3D imaging ability of our LFME system by imaging some randomly distributed fluorescence beads. We calculate and plot the lateral and axial MIPs of beads in Fig. 1(c). Specifically, we can identify different z positions (with ∼10.35μm depth distance) of two beads in Fig. 1(c), with the zoom-in image labeled by the green box below. As shown in the bottom of Fig. 1(c), the resolved FWHM of a certain bead is 6.06 μm in the x direction (lateral resolution) and 35.40 μm in the z direction (axial resolution). The relatively low axial resolution is the main limitation of our method currently, which we guess is also caused by the aberration introduced by the lens pair and non-ideal PSFs of the small MLA in use. We think it can also be further optimized by PSF calibration or PSF engineering [33–36].

Previously, we have demonstrated the 3D imaging ability by imaging some fluorescence beads. In this section, we further exhibit the 3D imaging of some biological samples in Fig. 6. We compare different reconstruction results using both RL deconvolution and the dictionary learning method. As shown in Figs. 6(a1) and 6(a2), we present the recovered lateral (x−y) MIP image of HeLA cells with RL deconvolution and the dictionary learning method, respectively. The x−z and y−z slices of the sample (marked by white dashed lines in the x−y MIP image) are exhibited simultaneously. At the bottom of Fig. 6, we plot the line profile of two beads with different z positions (about 25.95 μm distance), exhibiting the 3D imaging ability of our method. In the remaining two columns of Fig. 6, we also show x−y MIP images and z-slice images of the skin tissue in two different areas. We can see from the results that using the dictionary learning method can reduce reconstruction artifacts, enhance imaging contrast, and improve the z-axis imaging quality of our method.

For practical use, we adopt the epi-illumination mode for fluorescence endoscopic imaging in this work, which means the illumination light will pass through the SMFB before being projected on the sample. Due to the coherence of the laser source in use, cross-talk among different fiber cores will introduce undesirable interference and speckles into the illumination, resulting in the degradation of the imaging quality. Therefore, after the laser source, we use a continuously rotating diffuser, driven by a motorized rotation stage, to weaken the spatial coherence of the laser source, as the optical system shown in Fig. 1(a).

In this section, we use experimental imaging of the fluorescent plate, HeLa cells, and skin tissue to demonstrate the effectiveness of the rotating diffuser in LFME, and the results are shown in Fig. 7. It should be noted that the images in the former three columns of Fig. 7 directly use the lens pair for acquisition without using the MLA, while the images in the last column use both the lens pair and the MLA (i.e., the input raw light-field measurement). As shown in the first column of Fig. 7, without the diffuser, the images are contaminated by undesirable interference patterns and speckles. By applying a diffuser, the interference pattern is suppressed, but the speckles still exist, as shown in the second column of Fig. 7. By rotating the diffuser continuously, we can eliminate the speckles of images (in the third column of Fig. 7), since the spatial coherence of the laser source is weakened. Finally, by further assembling the MLA, the raw light-field measurement can be captured (in the last column of Fig. 7), from which we can get reconstructed images with high resolution and clean backgrounds [Figs. 7(b5) and 7(c5)].

As we have mentioned before, in this work, we put the MLA between the SMFB and the imaging lens pair to avoid the information loss caused by the spatial sampling of fiber cores, which means the excitation light will also transmit through the MLA [Fig. 1(a)]. Fortunately, by comparing the captured light-field measurements (the last column of Fig. 7) with the reference images (the third column of Fig. 7), we can find that transmitting through the MLA will hardly change the light intensity distribution on the sample. Since the spatially sampling patterns introduced by fiber cores can be eliminated by the above-mentioned resample step, the light-field deconvolution process can be performed successfully, as the final imaging performance is shown in Figs. 7(b5) and 7(c5).

In summary, we develop LFME, a light-field micro-endoscopy technique, that can acquire 3D information in a compact epi-fluorescence endoscopic system with relatively low invasion (2-mm imaging probe). We utilize a dictionary learning method for deconvolution to mitigate the reconstruction artifacts and increase the imaging contrast. We demonstrate the snapshot 3D imaging performance of LFME through experiments of beads, HeLa cells, and a human skin section, preserving better than 6.20 μm lateral resolution within ∼333μm-diameter circular FoV and ∼24μm DoF. Traditional endoscopy captures 3D information by using complicated optomechanical designs or calibration methods. In contrast, we pack the 3D information as the light-field measurement with simple optics and reconstruct them. The LFME scheme could become a promising and reliable endoscopic tool with a compact system for high-speed 3D imaging of biological tissues in vivo.

Since the volume acquisition rate of LFME is only limited by the camera framerate, further improvement may rely on developing real integrated equipment of LFME rather than the proof-of-concept system in this paper for high-speed 3D in vivo observation. The performance extension could be implemented to meet the requirement of applications with targeted adjustment. It is straightforward to extend LFME to a larger FoV by using SMFB with more fiber cores. Furthermore, improving the optical design [33] or introducing the aberration into the PSF model [34–36] can help the LFME maintain the optimal lateral resolution in a much wider DoF and improve the axial resolution simultaneously. Consequently, we believe such improvements in imaging performance will bring advanced imaging capability with a compact optical structure to the broad endoscopic community, facilitating quantitative analysis in various applications such as observation of organs or brain activities.

Acknowledgment. The authors thank Dr. Xuemei Hu for discussions.