Endoscopy is an optical imaging technique that has been widely practiced across the globe since the nineteenth century. Image quality here is of utmost importance, both for examination and diagnosis. There are many parameters that determine optimal image quality. Resolution is one such important metric, critical for endoscopic practices^1-3. This metric quantifies details in frames that are otherwise not visible to the naked eye without proper processing. Resolution in an optical system is principally limited by its diffraction limit. Therefore, overcoming the diffraction limit and achieving super resolution of an optical system has been a very dynamic research field for the last two decades.

There are several super resolution imaging techniques practiced prevalently in the field of microscopy. Stochastic optical reconstruction microscopy (STORM) and photo activated localization microscopy (PALM) are single molecule localization-based methods that bring optical resolution down to sub-10 nm scales^{4, 5}. However, they suffer from very slow imaging speeds. On the other hand, stimulated emission depletion (STED), is faster and has a good video rate⁶. However, the field of view is limited due to its point-by-point scanning approach. In addition, it needs exceptionally high laser intensity and complex instrumentation. Although traditional structured illumination microscopy (SIM), can only improve the resolution by 2-fold compared to the diffraction limit, it has been widely applied in bioimaging^{7, 8}. This wide field method can be operated under low intensity and high speeds. Recently, high resolution SIM technologies have been demonstrated by introducing plasmonics and metamaterials into SIM^9-12. Many variant super resolution techniques have also been developed by the combination of multiple methods mentioned above.

Despite the remarkable success, most of the super resolution microscopy techniques cannot be directly extended to the field of endoscopy. In a standard white light endoscope (WLE), the hardware used includes imaging and illumination optics which must be well confined within the endoscope’s distal tip. This means only compact optics that does not affect the size of the distal tip is allowed. In addition, large field of view, large depth of field and focus-free optics is required in endoscopic applications. Although high resolution endoscopic methods exist, they have significant limitations. One of the prominent high-resolution techniques that has been clinically used in gastrointestinal imaging is called confocal laser endomicroscopy (CLE)^{13, 14}. The CLE works by illuminating the tissue with a laser source and detecting the reflected fluorescent light from the tissue through a confocal aperture. Analogous to confocal laser microscopy, CLE is a scanning technique with a narrow field of view, typically in micrometers^15-17. Here the depth of field can be precisely selected at a time, but it is challenging to obtain a stereoscopic image with high speed. Hence, a comprise is commonly encountered in resolution vs. the field of view and depth of field. This may result in higher rates of sampling errors which inevitably leads to undesired complications^{18, 19}. A limited field of view at the cost of good resolution could also entail increased screening and operational time, cost, and a potential risk to the patient^20-24. To overcome these limitations, we need a robust method that can achieve high resolution simultaneously with a large field of view and depth of field.

Compared to other super resolution methods, SIM has the best potential to be applied in endoscopy due to its wide field and high-speed nature. In traditional SIM, known spatially structured patterns of light, like periodic fringes, are used to excite the sample. The spatial frequency mixing of the object and the illumination patterns occurs. Spatial mixing encodes structural details corresponding to the high spatial frequency of the sample into detectable low spatial frequency Moiré fringes^25-28. Therefore, high frequency components which are otherwise lost by conventional imaging techniques can be effectively recovered. A super resolution image of SIM is reconstructed through an image reconstruction algorithm using a series of low-resolution frames with different illumination patterns. The resolution of SIM can be determined by an effective numerical aperture (NA) given by

where NA_ill and NA_det represent the effective NA of illumination and detection optics, respectively. Traditional SIM utilizes the same optics for illumination and detection. This results in a resolution improvement of about twofold.

However, traditional SIM technologies emphasize absolute resolution so that optics with very high NA is used. This inevitably leads to very small depth of field which is not suitable for endoscopy. Additionally, well-defined illumination patterns in standard SIM can only be well predicted on a planar object plane. They become distorted, blurred, and unpredictable when they are projected onto an arbitrary and unknown 3D surface. This results in the loss of known illumination patterns. Recovering the image without prior knowledge of the illumination pattern can be overcome by using blind-SIM algorithm with a constraint that all the illumination patterns add up to a uniform intensity. This concept was demonstrated and called the speckle illumination technique^29-34. Speckle SIM is highly tolerable to illumination distortion as it is based on the statistical nature of the speckles. If we can separate the illumination and the image optics, with a fixed imaging system, the resolution would be primarily limited by the speckle resolution, thus, the overall speckle SIM resolution improvement factor may go far beyond twofold.

In this work, we propose and demonstrate a novel super resolution endoscopic technique called speckle structured illumination endoscopy (SSIE) to overcome the limitations described. This modality is aimed to achieve wide field, high resolution, and large depth of field in images. Unlike CLE, we do not make any modifications to the existing WLE’s image collection hardware, rather, we introduce an external light source from which the illumination patterns are transmitted and routed via multimode fibers onto the target. The advantage of using WLE’s imaging optics is that it offers a larger field of view and depth of field compared to CLE. The super resolution capability of the SSIE is brought about by the blind-SIM algorithm that makes use of the high-resolution speckle patterns projected onto the sample, hence, eliminating the dependence of resolution on the imaging optics of the endoscopic system. We prove the SSIE concept using fluorescence imaging on both 2D and 3D surfaces. Depending on the image distance, the SSIE resolution shows 2–4.5 times improvement compared to a standard WLE modality.

In this section we describe the working principle of SSIE. A standard surveillance WLE has distal and proximal ends with two main integral elements, namely the illumination source and the acquisition system. The illumination source in WLE is traditionally comprised of either lamps or LEDs which are routed from the proximal end to the distal end through the fiber bundles and onto the target³⁵. The acquisition system typically placed at the distal end, contains a couple of channels which include the air, water and biopsy ports, and the imaging optics which consists of a wide-angle objective lens and the image sensor³⁶. In SSIE, we retain all the components of a WLE, but add two multi-mode fibers along the insertion tube from an external low power laser. The idea here is to create a system where the clinician can switch between the incoherent illumination mode of the WLE and the coherent illumination mode of the SSIE as needed. The schematic of SSIE is shown in Fig. 1 emphasizing the additional components to a traditional WLE system.

In SSIE, a continuous wave (CW) low power laser (can be any wavelength but we select 532 nm as an example in this work) is coupled into a fiber collimator (objective of fiber launch: 40×/0.65NA). This optical field is directed into a fiber beam splitter via a speckle generator (e.g., a length of fiber with a vibrational motor). The speckle is controlled by the vibrational motor attached to the fiber spool which will stretch the fiber and change the interfering phase during image acquisition (see Supplementary information for more details). The fiber beam splitter couples the light into two identical multi-mode fibers surrounding the endoscope tube as shown in Fig. 1. The incident optical field is scrambled due to the multi-mode interference and can be changed by adding vibration to the fiber. The two multimode fibers are placed on either ends of the endoscope so that large angle interference of random speckles is formed containing high spatial frequency components beyond the diffraction limit of the image collection optics. The high-resolution information of the object is detected through frequency mixing of the object and the illumination speckles. Multiple frames of the object with these high-resolution speckles are captured and used for the final image reconstruction.

Based on the SSIE configuration shown in Fig. 2(a), we first simulated two exemplary speckle illumination patterns and their Fourier transforms which are depicted in Fig. 2(b) and 2(c). Clearly, the speckle resolution decreases with the distance. As the two fibers are placed along the y axis, the interference pattern along the y axis also shows slightly higher resolution.

As illustrated in Fig. 2(a), the two fibers are tilted towards the optical axis of the endoscope to ensure sufficiently large overlap for the interference area and cover the entire field of view (FOV) of a WLE at various focal depths. Since the NA of the fibers can be selected to be much larger than that of the detection optics, it is fairly easy to identify an appropriate tilt angle of the fibers, so that the interference area may fully cover the whole endoscope’s field of view. Assuming P(x₂, y₂, d) is an arbitrary location in the object plane, the spatial frequency of interference pattern at P is determined by 2k0sinα/2, where k0 is the wavevector of the laser light in free space, and α is the angle between the two-interfering light (see Fig. 2(a)). The angle α can be expressed as:

where w is the width of the endoscope. Applying the paraxial approximation, i.e., d>>x₂, y₂, the above equation becomes α=arccos ( 4d2−w24d2+w2)= arccos ( 1−2w24d2+w2). From Eq. (2), and its simplified form, we observe that α drops with an increase in distance d, which indicates closer the sample to the endoscope, more of the high frequency components of the illumination patterns will be utilized. We measured the speckle patterns using a CCD camera at different distance d, and the deduced and extrapolated effective illumination NA is summarized in Fig. 2(d). These results agree reasonably well with our numerical simulations, and the slightly low values in experiment are likely due to the low dynamic range of the CCD camera. To identify the detection NA (NA_det) in accordance with Eq. (1), we calibrate the endoscopic system against the standard USAF (United States Air Force Target) target (see more information in the Supplementary information). The experimentally calibrated values of NA_det are in Fig. 2(e). The SSIE resolution enhancement factor compared to traditional WLE is given by

and the estimated values are presented in Fig. 2(f), where a resolution enhancement of 2-4.5-fold is achievable. The placement of the fibers enables the large angle interference of the modulated speckle patterns which produces a high NA than that of the endoscopic detection NA, surpassing its diffraction limit as shown in Fig. 2(d, e). Notice the slightly different enhancement factors along x, and y axis, which is consistent with the asymmetrical Fourier spectra shown in Fig. 2(b) and 2(c). Such large resolution enhancement factors originate from the fine interreference illumination patterns. Using a single fiber source without interference will result in the NA_ill to drop by an order of magnitude (Supplementary information), which will in turn utterly diminish the resolution enhancement. As clearly shown in Fig. 2(f) and noted from the Eqs. (2) and (3), closer the sample to the endoscope’s distal end, higher will be the resolution enhancement.

To demonstrate the high-resolution capability of the proposed SSIE, we image sparsely distributed fluorescent stains (Rhodamine 6G) drop-casted on a glass slide. As from the working principle of the SSIE, the speckle illumination patterns are sensitive to the distance from the sample surface (Fig. 2(d)). In this section, we experimentally verify the performance of SSIE on a 2D surface at two exemplary chosen distances of 3.7 cm and 7.2 cm. First, we depict the results obtained at focal distance of 3.7 cm (Fig. 3). The process is as follows; at the sample plane high-resolution speckles excite the fluorescent stains in the specimen. The fluorescence signal is further collected directly by the endoscopic image sensor after passing through a 580/40 nm band pass filter, placed in the imaging path. Thereafter, the super resolution image is reconstructed by using the blind-SIM algorithm. Later, for comparison, the SSIE reconstructions are compared with the standard endoscopic image without the speckle structured illumination.

In this demonstration, 120 frames of images with various speckle illimitation patterns are collected. The diffraction limited endoscopic image is obtained by summing all the 120 frames shown in Fig. 3(a). This ensures the comparison is performed with the same imaging optics. The high resolution SSIE image after the blind-SIM reconstruction is shown in Fig. 3(b). The field of view in the SSIE image as captured by the endoscopic sensor is in millimeters (Fig. 3(b)). This is orders higher than CLE’s achievable limits which is typically in micrometers¹⁷. As expected, the SSIE image shows greatly improved resolution, so that the fine features including boundaries and edges are clearly resolved. As comparison, the stains are blurred with barely discernable edges in the diffraction limited image obtained by a standard WLE. To quantify the resolution enhancement factor of the SSIE, we performed Fourier transforms (FT) of the images in Fig. 3(a) and 3(b) and the results are shown in Fig. 3(c) and 3(d). Using Eq. (3), we obtain a resolution enhancement factor of 3.5 times and 3.39 times in the vertical direction (y) and the horizontal direction (x), respectively. This result is consistent with the expected enhancement values as shown in Fig. 2(f). Figure 3(e–g) show the zoom-in view of the object, diffraction limited image of the WLE, and the SSIE image. The resolution improvement and the sharp edges in the SSIE image is clearly visible. The optimal parameters and the number of iterations pertaining to the reconstruction makes the reconstruction accurate and the image features well defined.

From an information theory point of view, at least N² sub frames are needed to reconstruct super resolution image with N- fold resolution improvement. Traditional SIM uses well-known sinusoidal patterns to sample the object with high efficiency. Therefore, the oversampling factor can be very small, i.e., the total number of sub frames αN², where α, the oversampling factor, is close to 1. However, speckle-based blind-SIM requires more frames, i.e., α>1, owing to the lack of knowledge of the exact illumination patterns used and the correlations between the illumination patterns. By selecting a reasonableα, the speckle-based blind-SIM is known to produce high levels of resolution indicating its robustness⁹. We explore the SSIE’s performance for a secondary focal distance at 7.2 cm with 80 frames. The objective is to measure the results at a secondary focal distance to establish consensus in the results, keeping in mind that a typical endoscope has large depth of field.

The sample preparation is identical to the prior case. The enhanced image’s FT (Fig. 4(d)) yields a resolution enhancement of 2.5 times and 2.39 times along vertical (y) and horizontal (x) direction, respectively, compared to its diffraction limited equivalent (Fig. 4(c)). These enhancement factors also agree well with the expected values shown in Fig. 2(e), i.e., 2.52 and 2.4 times in the y and x directions, respectively. Even with a reduced number of frames, the SSIE image (Fig. 4(b)) is well resolved, which demonstrates the robustness and reliability of the speckle approach.

The standard WLE’s are typically used for examination of the human body where the surfaces imaged are not planar, hence, we consider imaging a 3D curved nonplanar surface. One advantage of using speckle SIM is that the projected patterns remain speckles when subjected onto a 3D plane after various distortions. Because the SSIE does not need to know the illumination patterns, the reconstruction remains unaffected. Rhodamine 6G is drop casted on a curved slide as indicated in Fig. 5(a) and used as the test sample for imaging.

A set of 60 frames were used in the SSIE reconstruction. The imaging distance is around 7.2 cm. The depth of field of the sample is about 3 mm. Figure 5(b) indicates a zoom in view of the ground truth wherein the feature edges are well defined. In the diffraction limited endoscopic image (Fig. 5(c)), we observe the features are blurred, unclear and appear as a blob of a stain. This is resolved well in the SSIE image (Fig. 5(d)) at wide field of view and depth of field (see Supplementary information for more information). Since the depth of the sample varies the average resolution improvement factor is ~2.52 times in the vertical direction and ~2.2 times in the horizontal direction, which is also reasonably consistent with Fig. 2(f) for imaging distance of 7.2 cm. If we compare the two experimental results in Fig. 4 and Fig. 5, we notice the similar imaging depth, i.e., ~7.2 cm, but with a difference in number of frames (80 and 60). The resolution enhancement is comparable between the two, indicating the oversampling factor α can be reduced to less than 10 without significantly affecting the image quality. Additionally, this example demonstrates that the SSIE can be directly applied to a clinical endoscope without adjustment of focusing, where the depth of field may go up to a few millimeters. Here, the depth of field is about 3 mm, but we anticipate the depth of field can be extended much further given the endoscope’s large depth of field. The prior demonstration proves and validates the concept of SSIE for 3D nonplanar surfaces where field of view and depth of field are both critical parameters along with enhanced levels of resolution. In the future, efforts will be directed into implementing the SSIE in a clinical setting, to quantize the imaging performance against the standard WLE.

The SSIE demonstrations are independent of the internal structure, type, or specifications of the scope. Therefore, the SSIE technique can be translated into any white light endoscopic modality with similar resolution improvement factors whether its application is clinical or industrial, since the working principle remains the same (see more information in the Supplementary information, Table S1). The imaging speed of the SSIE can further be enhanced by the employment of the high-speed spatial light modulator (SLM) to control the speckle patterns (see more information in the Supplementary information). Blind-SIM reconstruction algorithm is chosen over the conventional SIM due to its robustness to the deformations of the illumination patterns which is especially important in the case of endoscopic application. The time taken for imaging is about 3 minutes for a maximum of 120 frames used in this study. Frames were acquired at a rate of one frame per second as a few milliseconds within the second timeframe are accounted for the speckle pattern to settle after the multimode fiber has been modulated. This iterative algorithm may not be fast enough for real time image reconstruction, and we use it here mainly to demonstrate the image resolution enhancement for large depth of field focus free imaging. The computational efficiency of the iterative algorithm may be optimized by employing faster GPU's. In addition, future developments along the directions of neural networks may aid in making the SSIE more inclined towards a real time application, but its tradeoffs remain to be explored^37-39.

In conclusion, we propose and demonstrate the SSIE, a robust, wide field and low-cost super-resolution method for applications in endoscopic practices. The concept proof experiment was conducted with a standard WLE for image acquisition and two external multi-mode fibers for delivering the speckle illumination patterns onto the subject. The performance of the SSIE being mainly limited by the illumination NA which decreases over distance from the endoscopic image sensor is seen to produce an enhancement of 2–4.5 times the WLE equivalent based on the focal distances considered. SSIE greatly extends the spatial resolution with a cost of increased number of measurements. However, the time for data acquisition can be greatly reduced by implementing high speed imaging (Supplementary information). Since the endoscope requires no focusing, imaging objects with significant depth becomes rather straightforward and the speckle-based approach compliments the reconstruction efforts against possible distortions as otherwise is observed when utilizing traditional SIM against such objects. With respect to the setup, the SSIE is greatly simplified compared to other high definition or ultra-high-definition endoscopy techniques. Additionally, the SSIE does not rely on any specific properties of the specimen or sample, therefore any sample can be used. In the future, we anticipate that the SSIE will transform the standard white light endoscope into a much more accurate imaging tool, leading to profound benefits to the GI endoscopic community and the patient.

The experimental demonstration of the SSIE took frames equal to and lower than 120 (1 frame per second) to generate data’s (Figs. 3, 4, 5). The fluorescent signal is collected by the endoscopic lens with a band pass filter (580/40 nm). The wide field of view image reconstruction takes about three minutes on a desktop computer with GTX 1080Ti graphics card and a i7-8700k. All experiments were conducted by using the white light Teslong’s Borescope Endoscope. The dye used was Rhodamine 6G which was diluted against ethanol at 8 mg/mL concentration. This is drop casted on a glass slide using a pipette. The US air force target utilized was that of the Edmunds USAF resolution target, pocket size. The microscope used to establish ground truth was, Olympus IX83, with 50× magnification, 0.5 NA and Olympus SZ40, with the eyepiece 20×, objective: zoom lens 0.6-4× (stereo microscope). The multimode fibers used were from Thorlabs (0.22 NA, High-OH, 50 μm core, Lambda: 250–1200 nm, FC/APC termination). The laser utilized is the Newport, continuous wave, power 200 mW and the corresponding neutral density filters used in the light path was from Thorlabs (lower than two orders of attenuation). The fiber beam splitter used was from OZ optics (50/50 beam split ratio).