Three-dimensional (3D) imaging has been a long and actively pursued technology due to its important applications in medical diagnosis [1–3], remote sensing [4,5], facial recognition [6–8], environmental monitoring [9,10], and so on. A handful of techniques and realizations have thus far been demonstrated, including those based on structured-light imaging [8,11], light detection and ranging (LiDAR) with raster scanning [6,7,12,13], and stereophotogrammetry [14]. Recently, the time-correlated single-photon counting technology has been deployed to boost the detection sensitivity [15–17]. In general, those active-illumination systems can generate 3D profiles of a target object with higher accuracy than those based on passive sensing.

Meanwhile, infrared (IR) imaging and detection have been studied extensively in the past decade, which allows sensitive detection of many biomolecular and chemical signals [18,19] compared to that of visible light [20,21]. However, existing IR detection techniques are mostly based on thermal sensors, which have low sensitivity and high noise, even with cryogenic cooling [22]. On the other hand, visible detectors have much lower noise and high sensitivity without the need for cryogenic cooling. Instead of direct IR imaging, parametric frequency upconversion imaging [23–25] plays a critical role for hyper-spectral IR imaging, where the IR signal is frequency upconverted into visible wavelength [20,26–29] and detected by a silicon-based detector or camera with high sensitivity and low noise. Many unique nonlinear optical systems have also been developed for two-dimensional (2D) imaging, such as noiseless optical parametric amplification imaging [30,31], non-degenerate two-photon absorption [32–34], and spontaneous parametric downconversion imaging [35,36]. Some are deployed to NIR or MIR imaging regimes. With parametric frequency upconversion, it has been shown that near-unity conversion efficiency can be achieved, and also can preserve the quantum characteristics of IR photons [37]. This will facilitate near-IR (NIR) or MIR imaging at a few photon levels with low dark noise [20], for example, by utilizing a sensitive detector such as silicon electron-multiplying charge-coupled device (EMCCD) to directly record upconversion photons in the visible or NIR region [24,25].

The natural extension of the existing 2D parametric upconversion imaging system to 3D imaging can have great potential and may offer promising applications that require IR multidimensional imaging. One attempt of 3D IR imaging with a parametric upconversion process is shown in Ref. [38]. Here they used a chirped ultrashort pulse as the laser source and utilized the principle of ultrafast conversion between space, time, and frequency to obtain a single shot classical 3D image. However, the weakness of their study was due to the trade-off between the measurement range and depth resolution, as well as the low spatial resolution due to tight focusing at the crystal plane. This makes it difficult to extract critical features of the objects.

Some common challenges in most of the backreflected imaging systems are strong background noise and multiple scattering [33,39–41]. In order to strip the background noise from the contaminated signal, traditional methods such as time-frequency filters are commonly used [42,43], but they are inherently limited by the trade-off between signal detection efficiency and noise rejection. Realizing this, quantum parametric mode sorting has been proposed and demonstrated [43–45]. Some other optical techniques are also present in the literature to address imaging through multi-scattered media. Most of these techniques can be broadly classified as relying on ballistic photons [39,46], diffuse optical tomography [47,48], etc. Those techniques offer potential applications for medical imaging, communications, and security [2,49]. However, most studies in the field of overcoming strong background noise and multiple scattering have focused on 2D imaging. In the past, quantum parametric mode sorting has been utilized for single-pixel 3D single photon imaging systems in photon-starved, noise-polluted environments as well as imaging through a strongly scattering medium [7,50]. With raster scanning methods, it demonstrated imaging of a 3D object through a highly reflective obscuring scene with 36 dB advantage in noise rejection. However, the requirement of raster scanning has severely impeded the image acquisition speed while the traverse spatial resolution of the 3D image is limited by the field of view of the single-pixel detector [51].

In this paper, we extend these promising studies and explore an active NIR 3D imaging system using EMCCD with upconversion detection through a highly scattered medium amid strong spatiotemporal background noise. Our system is based on the nonlinear frequency upconversion process via sum-frequency (SF) generation of time-correlated pump and signal optical pulses. Combined with single-photon sensitive EMCCD, we capture spatial and temporal information of the scene of interest, which can be used to reconstruct 3D surface profiles of the target object. To surmount the imaging through a strong scattering medium, we utilize time-resolving photon detection by detecting the frequency upconversion image confined temporally within the time window of the pump pulse, which shows excellent behavior in noise rejection. Compared with the raster scanning methods [7,12,13], in our work, we could significantly improve the spatial resolution up to 48 μm and effectively reduce the 3D image acquisition time. Besides, we could reconstruct the 3D profile of the target object through strong spatiotemporal background noise (SNR about −20dB). This system can be deployed in applications that require ultra-sensitive imaging, such as medical diagnostics and quantum optics at the single- or few-photon level [52,53], and it can find values in biomedical imaging, non-destructive label free diagnosis, and quantum communications. In the future, selective 3D edge enhancements can be implemented by imprinting the spatial phase patterns on the pump beam using a spatial light modulator (SLM) [54,55].

The nonlinear optical setup for the 3D image reconstructor is shown in Fig. 1(a). The signal and pump pulses are derived from a 50 MHz femtosecond mode-locked laser (MLL) using two inline narrowband wavelength division multiplexers (WDM) of 0.8 nm linewidth to pick two separate wavelengths, one at 1545 nm as the signal and another at 1558 nm as the pump. The pump pulses are sent to a programmable optical delay line (ODL) and then guided into free space with collimated beam size of 0.65 mm FWHM. The collimated signal beam is magnified by a beam expander to 10.8 mm FWHM. The intensity of the horizontally polarized signal beam is tuned by the combination of half- and quarter-wave plates (HWPs and QWPs) with a polarizing beam splitter (PBS). The signal light, after PBS, then passes through another QWP and scattering media before incident on the object. The backreflected or scattered light from the object then changes the polarization and passes through a telescope with lenses FL1 (focal length 300 mm) and FL2 (focal length 25 mm), which reduce the beam size to ∼0.53mm FWHM. After that, a beam splitter (BS) combines the collimated signal and pump beams that are incident into a temperature-stabilized PPLN crystal with the poling period of 19.36 μm [5% (mole fraction) MgO doped PPLN, 2 mm length, 3 mm width, and 1 mm height from HC Photonics] for the frequency conversion process. The normalized conversion efficiency in our case is 9×10−4%/W, which is restricted by three factors: (a) low pump power, (b) short crystal length (2 mm), and (c) being not in the optimum focusing condition for signal and pump (both are collimated beams) inside the crystal. The 4f system after the crystal consists of two Fourier lenses with focal length 25 mm (FL3) and 100 mm (FL4), imaging the SF output onto an EMCCD (iXon Ultra 897, Andor) with 512×512 pixels and 16 μm pixel size. The quantum efficiency of this EMCCD is measured to be 7.5% at upconversion wavelength, calibrated against a silicon avalanche photodiode at the mean photon number ≈0.01 per pulse. On the other output of the BS, an IR camera (FIND-R-SCOPE Model No. 85700) is used to capture the image via direct signal detection.

First, we evaluate the transverse spatial resolution of the 3D imager with two different experimental settings, as shown in Fig. 2. A 1951 USAF resolution test chart (USAF-RTC) is used at the object position to test the spatial resolution of this system. There are 54 target elements provided in the USAF-RTC, and each element consists of three bars that are separated by the bar width. We define the feature size as the width of the bar in the USAF-RTC, which is half the distance between the centers of the two bars. The spatial resolution result shown in Fig. 2(a) is the upconversion image of group 1 in USAF-RTC, which was obtained from the experimental setup depicted in Fig. 1(a). The intensity versus position line plot shown in Fig. 2(b) reflects the one-dimensional spatially resolved blue dashed line in Fig. 2(a). It shows that our setup is able to easily resolve features with size ∼144μm, which is also consistent with its actual value 140.31 μm (group 1, element 6). The decrease in the intensity at the edges is due to the Gaussian intensity distribution of the probing signal beam and the pump beam. We have validated that the size of the spot hitting on the object and the subsequent imaging system affect the final resolvable resolution of our 3D imager system since they affect the point spread function of the backreflected signal from the object. So, another version of the experiment setup has been constructed, whose spatial resolution result is shown in Fig. 2(c), which is the upconversion image of group 3 in USAF-RTC. In this case, the signal beam size is reduced to 3.6 mm FWHM. Besides, the Fourier lenses of the first 4f system are replaced as FL1 (focal length 100 mm) and FL2 (focal length 25 mm). The intensity profile along the red dashed line in Fig. 2(c) is shown in Fig. 2(d), where we can easily resolve objects with spatial resolution 48 μm. This feature size is matched with its actual value 49.37 μm (group 3, element 5). In this case, the resolvable feature size is improved, but the field of view from the observed object is reduced. In order to image bigger objects, we present the following results in this work using the setup shown in Fig. 1(a), with the spatial resolution of 144 μm.

Our 3D object image reconstruction technique utilizes the spatial and temporal photon information enabled by the nonlinear frequency conversion process. Figure 3 shows an example of the 3D reconstruction process. In Fig. 3(a1), the blue curve indicates the normalized photon intensity as a function of the arrival time for the reflected/backscattered signal photons from the object to reach the crystal, and the red solid curve represents the pump pulse arrival time. By sweeping the ODL, the pump pulses are swept along the temporal domain, which can overlap with the returning signal pulses at a certain arrival time. These two overlapped pulses interact inside the PPLN crystal, and the SF light is generated as shown in Fig. 3(a2). The signal can be upconverted efficiently only if it is spatially and temporally overlapping with the pump. At each delay step, the SF image was captured to reconstruct the depth z (=c×t/2) of the 3D object. Therefore, the optical delay time indicates the arrival time of the backscattered signal photons. In Figs. 3(a3)–3(a5), three SF images were collected at optical delay time t=75ps, t=105ps, and t=140ps, respectively. The collected data are a 3D data set shown in Fig. 3(a6). Each pixel has one histogram of the photon’s arrival time versus the counts. After converting the photon’s arrival time into distance, we show two histograms for two different pixels in Fig. 3(a6). The peak value (z1 and z2) of the curve provides the information of relative depth for each pixel on the object. By post-processing the data, the reconstructed image with depth information is shown in Fig. 3(a7) followed by applying a median filter (4×4 pixels) to smoothen the reconstructed image. In the experiment, a washer and a bolt are used as the target objects to perform the measurement. The dimensional measurements of the washer and the bolt are shown in Figs. 3(b1) and 3(c1), respectively. The results of the reconstructed 3D images are shown in Figs. 3(b2) and 3(c2). The X axis and Y axis give the cross section of the target object in millimeters, and Z axis shows the depth information of the target object in millimeters. In Fig. 3(b2), the measured outer diameter of the reconstructed washer is 7.92 mm, the inner diameter is 3.65 mm, and the depth is 1.6 mm. In Fig. 3(c2), the measured stub height and diameter of reconstructed bolt are 5.9 and 2.3 mm, and the height and the diameter of the bottom parts are 3.2 and 6.45 mm, respectively. We set the EMCCD exposure time of each image to 1 s. The total acquisition time for a full 3D image reconstruction is ∼30s. The reconstructed 3D images in Figs. 3(b2) and 3(c2) agree well with the ground truth shown in Figs. 3(b1) and 3(c1).

Next, we test the performance of our technique through scattering media. The scattering media are made from epoxy resin and titanium oxide (TiO2) pigment (∼220nm particle size). We examine two different pieces of scattering media (SM1 and SM2) in our setup. The thickness, mean free path (ls), and optical depth of the scattered media are shown in Table 1. SM1 with thickness 4.3 mm and optical depth 7.29ls (14.58ls round trip) is more scattered compared to SM2 with thickness 3.2 mm and optical depth 4.04ls (8.08ls round trip). In our case, the scattering media are placed in front of the object, so the signal propagates twice through the scattering media before upconversion detection. The backreflected photons coming from the different surface of the target object are upconverted in different time intervals using the pump pulse time gating. It could allow us to isolate the other backscattered noise photons arriving in different time intervals. Figure 4 shows the 3D image reconstruction of the target objects, bolt [Figs. 4(a) and 4(c)] and washer [Figs. 4(b) and 4(d)] through scattering media (SM1 and SM2), respectively. To effectively reconstruct the 3D image, we used a time windowing procedure to post-select several continuous EMCCD images that are captured at different temporal delays of the pump pulse. The time window could discard most of the background noise coming from the scattering and process only those within the time window defined by the pump pulses, following the procedure shown in Figs. 3(a6)–3(a7). This procedure is programmed via MATLAB to improve the SNR of the reconstructed 3D image. It reduces the speckle noise induced by the scattering medium and, thus, better resolves the shape of the target object. The reconstructed images of the bolt through SM1 without and with time windowing are shown in Figs. 4(a1) and 4(a2), respectively. Here, the edge and depth variation of the cap bolt can be distinguished clearly. The washer images without and with temporal windowing are also reconstructed through SM1 in Figs. 4(b1) and 4(b2), respectively. It can also reconstruct the image of the washer. Similar results are shown in the third row of Fig. 4 for relatively weaker scattering media (SM2). In both cases, we can effectively reconstruct the 3D image of the target object by carving the temporal window.Table 1.

Parameters of Scattering Media

After that, we inject the amplified spontaneous emission (ASE) noise that is temporally and spectrally overlapping with the signal, as shown in Fig. 1(a). This noise is generated from an erbium-doped fiber amplifier (EDFA). Both the signal and ASE noise pass twice through SM2 before the upconversion detection. To ensure that the ASE noise is in the same time-frequency and spatial profile, we choose the same WDM filter bandwidth and spatial beam size as the signal. In this measurement, we use the washer in Fig. 3(b1) as the target object. Figure 5(a) shows the image of the signal mixed with the temporal noise before upconversion, as taken by the IR camera. At a low SNR (about −20dB), it is impossible to reconstruct the image of the target object with direct detection. Yet, our system captures an image with up to 8×107 converted photons per second using the EMCCD as shown in Fig. 5(b). By time windowing, the noise is effectively suppressed, and the reconstructed washer image gives ∼16dB improvement in the SNR. Figure 5(c) shows the 3D reconstruction image of the target object by temporally scanning the pump pulses. The exposure time of the EMCCD is the same as the other previous measurements (=1s). This result clearly demonstrates the noise rejection advantage of our system.

We now briefly discuss the effects of the EMCCD exposure time on the reconstructed 3D images. Figure 6 shows the results for three different exposure times: 0.1 s, 1 s, and 2 s. As the exposure time increases, the 3D image of the washer can be easily recognizable with better contrast. For the exposure time of 2 s on EMCCD, the total acquisition time in reconstructing a 3D image is ∼60s, which is considerably long. Yet, as the above improvement comes from the increased number of photons collected by the EMCCD, one can instead increase the nonlinear conversion efficiency or use detector arrays with higher quantum efficiency to enhance the image contrast while reducing the acquisition time.

Thus far, we have used highly reflective metal objects as our testing targets. To further assess the capability of our photon sensitive 3D imager for general objects, we now switch to a target with a diffusive surface, shown in Fig. 7(a). This diffusive surface, a digit “3,” is homemade using PLA plastic filament by a 3D printer. For 3D reconstruction of this object, we set the EMCCD exposure time to 2 s. Figure 7(b) shows the reconstructed 3D image without any scattered media or ASE noise. It shows that the 3D reconstructed image is clearly recovered. When we add the same scattering media and ASE noise as used in Fig. 5, our 3D imager performance degrades. Nonetheless, we are still able to recover the object by properly choosing the temporal window. Figure 7(c) presents the results with the full time scanning, while Fig. 7(d) gives the manicured data by carving the temporal window. In these cases, we need to increase the EMCCD exposure time to 10 s to collect more photons. As seen, the unrecognizable 3D image in Fig. 7(c) can be well recovered in Fig. 7(d) by properly tuning the temporal window.

We have experimentally demonstrated a high-performance 3D imager for photon sensitive detection using optical frequency upconversion pumped by picosecond pulses. It obtains millimeter depth resolution and 140 μm spatial resolution, while effectively rejecting background noise from ambient environment and obscurants. As such, the present technique could potentially find applications in biomedical imaging, remote sensing over low visibility. On the other hand, in the current method, the acquisition time of photon sensitive imaging is longer than what is needed for typical real-time target object identification. This shortcoming is mainly due to the low conversion efficiency of the current nonlinear process, which can be increased by using a high-power laser or a longer nonlinear crystal. Also, to improve the imaging sensitivity through different kinds of scattering materials, one could use spatially modulated pump beams to further improve the 3D image contrast, similar to what has been demonstrated in 2D imaging cases [54,55].