2D optical visualization of transient phenomena in the actual time of the event’s occurrence plays a vital role in the understanding of many mechanisms in biology, physics, and chemistry [1–3]. To discern spatiotemporal details in these phenomena, high-speed optical imagers are indispensable. Imaging speeds of these systems, usually determined by the frame rates of deployed CCD or CMOS cameras, can be further increased using novel sensor designs [4–6], new readout interfaces [7,8], and advanced computational imaging methods [9–11].

Among existing approaches, compressed ultrafast photography (CUP) [12–20] is an innovative coded-aperture imaging scheme [21,22] that integrates video compressed sensing [23] into streak imaging [24]. In data acquisition, a spatiotemporal $(x,y,t)$ scene is compressively recorded by optical imaging hardware to a 2D snapshot. The ensuing reconstruction computationally recovers the datacube of the scene. Despite initially demonstrated using a streak camera, the concept of CUP was soon implemented in CCD and CMOS cameras in compressed optical-streaking ultrahigh-speed photography (COSUP) [25]. Compared to other single-shot ultrahigh-speed imaging modalities [26–29], COSUP is not bounded by the moving speed of piezo-stages [26,27] or the refresh rate of spatial light modulators [28,29]. As a cost-efficient system, COSUP has demonstrated single-shot transient imaging ability with a tunable imaging speed of up to 1.5 million frames per second (fps) based on an off-the-shelf CMOS camera with an intrinsic frame rate of tens of hertz.

Despite these hardware innovations, COSUP’s video reconstruction has ample room for improvement. Existing reconstruction frameworks can be generally grouped into analytical-modeling-based methods and machine-learning-based methods [30]. Using the prior knowledge of the sensing matrix and the sparsity in the transient scene, the analytical-modeling-based methods reconstruct videos by solving an optimization problem that synthetically considers the image fidelity and the sparsity-promoted regularization. However, demonstrated methods, such as the two-step iterative shrinkage/thresholding (TwIST) algorithm [31], augmented Lagrangian algorithm [32], and an alternating direction method of a multiplier (ADMM) algorithm [29], undergo time-consuming processing that uses tens to hundreds of iterations. The excessively long reconstruction time strains these analytical-modeling-based methods from real-time (i.e., $\ge 16\mathrm{Hz}$ [33]) reconstruction, which excludes COSUP’s application scope from tasks that need on-time feedback [34]. Moreover, the reconstructed video quality highly depends on the accuracy of prior knowledge and the empirical tuning of parameters.

To solve these problems, machine learning has become an increasingly popular choice. Instead of relying solely on prior knowledge, large amounts of training data are used for deep neural networks (DNNs) [35] to learn how to map an acquired snapshot back to a video. Upon the completion of training, DNNs can then execute non-iterative high-quality reconstruction during runtime. Thus far, DNNs that employ the architectures of the multilayer perceptrons (MLPs) [36,37] and the U-net [38–41] have shown promise for compressed video reconstruction. Nonetheless, MLPs, with fully connected structures, scale linearly with the dimensionality of input data [42]. Besides, the decomposition in the reconstruction process presumes that all information in the output video block is contained in a patch of the input image, which cannot always be satisfied [36,37]. As for the U-net, the reconstruction often starts with a pseudo-inverse operation to the input snapshot to accommodate the equality in dimensionality required by the original form of this network [43]. This initial step increases the reconstruction burden in computational time and memory. Moreover, akin to MPLs, U-net-based methods require slicing input data for reconstruction, which could cause the loss of spatial coherence [39]. Finally, inherent temporal coherence across video frames is often unconsidered in the U-net [44]. Because of these intrinsic limitations, videos reconstructed by the U-nets are often subject to spatiotemporal artifacts and a shallow sequence depth (i.e., the number of frames in the reconstructed video) [41].

Here, we propose a way to overcome these limitations using an autoencoder (AE), whose objective is to learn a mapping from high-dimensional input data to a lower-dimensional representation space, from which the original data is recovered [45]. The implementation of convolutional layers in AE’s architecture provides a parameter-sharing scheme that is more efficient than MLPs. Besides, without relying on locality presumptions, deep AEs with convolutional layers can preserve the intrinsic coherence in information content. Furthermore, recent advances in combining AE with adversarial formulations [46] have allowed replacing the loss functions based on pixel-wise error calculation to settings where perceptual features are accounted for, which have enabled a more accurate capture of data distribution and increased visual fidelity [47]. In the particular case of training generative models [e.g.,  generative adversarial networks (GANs)] for natural scenes, recent advances have improved the reconstructed imaging quality by dividing the overall task into sub-problems, such as independent modeling of the foreground and background [48], separated learning of motion and frame content [49], and conditioning generation on optical flows [50]. Despite these advances, with popular applications in audio signal enhancement [51] and pattern recognition [52], AEs have been mainly applied to 1D and 2D reconstruction problems [53,54]. Thus, existing architectures of AEs cannot be readily implemented for video reconstruction in compressed ultrahigh-speed imaging.

To surmount these problems, we have developed a snapshot-to-video autoencoder (S2V-AE)—a new DNN that directly maps a compressively recorded 2D $(x,y)$ snapshot to a 3D $(x,y,t)$ video. This new architecture splits up the reconstruction process into two sub-tasks, each of which is trained individually to obtain superior quality in reconstructed videos. Implemented in compressed ultrahigh-speed imaging, such a video reconstruction framework enables the development of a single-shot machine-learning-assisted real-time (SMART) COSUP, which is applied to tracking multiple fast-moving particles in a wide field at 20,000 fps (20 kfps).

The schematic of the SMART-COSUP system is shown in Fig. 1(a). Its operating principle contains single-shot data acquisition and real-time video reconstruction [Fig. 1(b)]. A dynamic scene, $I(x,y,t)$, is imaged by front optics onto a printed pseudo-random binary transmissive mask (Fineline Imaging, Inc., Colorado Springs, CO, USA) with encoding pixels of $25\mathrm{\mu m}\times 25\mathrm{\mu m}$ in size. This spatial modulation operation is denoted by the operator $\mathbf{C}$. The intensity distribution right after the encoding mask is expressed as $$I_{\mathrm{c}}(x,y,t)=\sum _{j,k}I(\frac{x}{M_{\mathrm{f}}},\frac{y}{M_{\mathrm{f}}},t)C_{jk}\mathrm{rect}(\frac{x}{d_{\mathrm{e}}}-j,\frac{y}{d_{\mathrm{e}}}-k).$$(1)

Here, $M_{\mathrm{f}}$ is the magnification of the front optics, $C_{jk}$ denotes an element of a binary matrix representing the encoding pattern, $j$ and $k$ are matrix element indices, $d_{\mathrm{e}}$ is the size of encoding pixels on the mask, and $\mathrm{rect}(·)$ represents the rectangular function.

Subsequently, the spatially modulated scene is relayed by a $4f$ imaging system that consists of a galvanometer scanner (GS, 6220 H, Cambridge Technology, Bedford, MA, USA) and two identical lenses (Lens 1 and Lens 2, AC254-075-A, Thorlabs, Inc., Newton, NJ, USA). The GS is placed at the Fourier plane of this $4f$ imaging system to conduct optical shearing in the $x$ direction, denoted by the operator ${\mathbf{S}}_{\mathbf{o}}$. The sheared image can be expressed as $$I_{\mathrm{s}}(x,y,t)=I_{\mathrm{c}}(x-v_{\mathrm{s}}t,y,t),$$(2)where $v_{\mathrm{s}}$, which denotes SMART-COSUP’s shearing velocity, is calculated by $v_{\mathrm{s}}=\alpha V_{\mathrm{g}}{f}_2/t_{\mathrm{g}}$. Here, $V_{\mathrm{g}}=0.16–0.64\mathrm{V}$ is the voltage added onto the GS, $\alpha$ is a constant that links $V_{\mathrm{g}}$ with GS’ deflection angle with the consideration of the input waveform, $f_2=75\mathrm{mm}$ is the focal length of Lens 2 in Fig. 1(a), and $t_{\mathrm{g}}=50\mathrm{ms}$ is the period of the sinusoidal signal added to the galvanometer scanner.

Finally, the dynamic scene is spatiotemporally integrated by a CMOS camera (GS3-U3-23S6M-C, Teledyne FLIR LLC, Wilsonville, OR, USA) to a 2D snapshot, denoted by the operator $\mathbf{T}$. The optical energy $E(m,n)$ measured at pixel $(m,n)$, is $$E(m,n)=\iiint I_{\mathrm{s}}(x,y,t)\mathrm{rect}(\frac{x}{d_{\mathrm{c}}}-m,\frac{y}{d_{\mathrm{c}}}-n)\mathrm{d}x\mathrm{d}y\mathrm{d}t.$$(3)

Here, $m$ and $n$ are the pixel indices in the $x$ and $y$ axes on the camera, and $d_{\mathrm{c}}=5.86\mathrm{\mu m}$ is the CMOS sensor’s pixel size. From Eqs. (1)–(3), the forward model of SMART-COSUP is expressed by $$E(m,n)=\mathbf{T}{\mathbf{S}}_{\mathbf{o}}\mathbf{C}I(x,y,t).$$(4)

In the ensuing real-time video reconstruction, the captured data is transferred to a workstation equipped with a graphic processing unit (RTX Titan, NVIDIA, Santa Clara, CA, USA). The S2V-AE retrieves the datacube of the dynamic scene in 60 ms. The frame rate of the SMART-COSUP system is derived from $$r=\frac{v_{\mathrm{s}}}{d_{\mathrm{c}}}.$$(5)

In this work, the reconstructed video has a frame rate of up to $r=20\mathrm{kfps}$, a sequence depth of $N_t=40–100$ frames, and a frame size of up to $N_x\times N_y=256\times 256$ pixels.

Compared to the previous hardware configuration [25], SMART-COSUP replaces the digital micromirror device (DMD), which functions as a 2D programmable blazed grating [55], with the transmissive mask for spatial modulation. This arrangement avoids generating a large number of unused diffraction orders, preventing a limited modulation efficiency to unblazed wavelengths, and eliminating intensity loss from the reflection from its cover glass as well as by its interpixel gap. In addition, the printed mask is illuminated at normal incidence, making it fully conjugated with both the object and the camera. Thus, the SMART-COSUP system presents a simpler, economical, and compact design with improved light throughput of the system and image quality of the captured snapshot.

The architecture of S2V-AE consists of an encoder and a generator [Fig. 2(a)]. The encoder (denoted as $\mathcal{E}$) converts a 2D snapshot to a series of low-dimensional latent vectors that represent particular features of the dynamic scene under study. As shown in Fig. 2(b), its architecture consists of five convolutional layers, a bidirectional long short-term memory (Bi-LSTM) recurrent layer [56], and a fully connected layer. In the convolutional layers, each convolution operation is followed by batch normalization (BN) [57] along with rectified linear unit (ReLU) activation [58]. The number of channels of feature maps, denoted by $N$, decreases from a preset value (512 in our experiments) to $N_t$. Then, the feature map, output by the last convolutional layer, is reshaped from a tensor into $N_t$ vectors, all of which are fed into the Bi-LSTM recurrent blocks with the fully connected layer to model temporal coherence. The outputs of the encoder, referred to as latent vectors, are then input to the generator (denoted as $\mathcal{G}$). In particular, each latent vector is reshaped back to a tensor, which is fed into the generator to reconstruct one frame in the video. As shown in Fig. 2(c), the architecture of the generator consists of seven transposed convolutional layers. Akin to the encoder, BN and ReLU activation are employed after each transposed convolution, whose preset number of channels decreases from 1024 to 1. Each latent vector is processed by the generator to a frame of $N_x\times N_y$ in size. The composition of $N_t$ such frames produces the reconstructed video.

The training of the encoder and the generator in the S2V-AE is executed sequentially. Training data are generated on the fly. The details of the training data collection and the training procedure are described in our open source code (see the link in Disclosure). Additional data, not included in its training phase, are used for evaluation. The generator is first trained under the setting of a GAN with multiple discriminators to ensure sufficient diversity. In brief, a random noise vector ${\mathit{z}}_{\mathrm{Noise}}$, sampled from a prior distribution $p_{{\mathit{z}}_{\mathrm{Noise}}}$ (i.e., ${\mathit{z}}_{\mathrm{Noise}}\sim p_{{\mathit{z}}_{\mathrm{Noise}}}$), is input to the generator to produce a fake frame $\mathcal{G}({\mathit{z}}_{\mathrm{Noise}})$ that is expected to have visual similarity with the real frame ${\mathit{x}}_{\mathrm{Data}}$ with an implicit data distribution $p_{{\mathit{x}}_{\mathrm{Data}}}$ (i.e., ${\mathit{x}}_{\mathrm{Data}}\sim p_{{\mathit{x}}_{\mathrm{Data}}}$). The fake or real data are judged by $K=40$ discriminators [Fig. 2(d)]. In each such discriminator, the data are first projected by a random matrix (denoted by $p_k$, where $k=1,2,\dots ,K$) to lower dimensions. Then, each discriminator (denoted as ${\mathcal{D}}_k$) converts the input to a number that is expected to be high for a real frame and low for a fake frame. Each discriminator, corresponding to a binary classifier as schematically shown in Fig. 2(e), consists of seven convolutional layers with the numbers of channels ranging from 1024 to 1. The loss functions of each discriminator $\{{\mathcal{D}}_k\}$ ($k=1,2,\dots ,K$) (denoted by ${\mathcal{L}}_{{\mathcal{D}}_k}$) and the generator (denoted by ${\mathcal{L}}_{\mathcal{G}}$) are calculated by $$\begin{gathered} {\mathcal{L}}_{{\mathcal{D}}_k}=-{\mathbb{E}}_{x_{\mathrm{Data}}\sim p_{x_{\mathrm{Data}}}}[\log ({\mathcal{D}}_k(x_{{\mathrm{Data}}_k}))] \\ {-\mathbb{E}}_{{\mathit{z}}_{\mathrm{Noise}}\sim p_{z_{\text{Noise}}}}[\log (1-{\mathcal{D}}_k(\mathcal{G}{({\mathit{z}}_{\mathrm{Noise}})}_k))], \end{gathered}$$(6)$${\mathcal{L}}_{\mathcal{G}}=-\frac{1}{K}\sum _{k=1}^K{\mathbb{E}}_{{\mathit{z}}_{\mathrm{Noise}}\sim p_{z_{\text{Noise}}}}[\log ({\mathcal{D}}_k(\mathcal{G}{({\mathit{z}}_{\mathrm{Noise}})}_k))].$$(7)

Here, ${\mathcal{L}}_{{\mathcal{D}}_k}$ corresponds to the cross-entropy loss [59]. After the random projection $\{p_k\}$, the input to each discriminator is either $x_{{\mathrm{Data}}_k}$ or $\mathcal{G}{({\mathit{z}}_{\mathrm{Noise}})}_k$. Note that each discriminator is trained on two mini-batches of samples (i.e., real frames and fake frames). The notations ${\mathbb{E}}_{x_{\mathrm{Data}}\sim p_{x_{\mathrm{Data}}}}[·]$ and ${\mathbb{E}}_{{\mathit{z}}_{\mathrm{Noise}}\sim p_{z_{\mathrm{Noise}}}}[·]$ indicate taking the expectations over the distribution $x_{\mathrm{Data}}\sim p_{x_{\mathrm{Data}}}$ and ${\mathit{z}}_{\mathrm{Noise}}\sim p_{z_{\mathrm{Noise}}}$, respectively. These loss functions are estimated over mini-batches to compute the gradients of losses for each parameter update. Moreover, training iterations are such that each discriminator is first updated in the descent direction of its corresponding loss and then ${\mathcal{L}}_{\mathcal{G}}$’s gradients are used to update the generator’s parameters. The described training game is expected to converge to equilibrium (i.e., no player can improve without changing the other player), which is not guaranteed to occur in highly non-convex cases, such as in the training of neural networks. However, the results found in practice in our setting are satisfactory. Successful training of the generator will yield parameters that enable its outputs $\mathcal{G}({\mathit{z}}_{\mathrm{Noise}})$ to resemble characteristics of the real data. Leveraging this architecture, the goal of each discriminator is to distinguish the real data from the fake ones. The generator, by contrast, aims to fool all discriminators by learning how to produce frames as close as possible to real data. Parameters in the generator and discriminators are updated according to these loss functions [i.e., Eqs. (6) and (7)], which are minimized via gradient-descent-based optimization.

As the second step, the encoder is trained with the parameters of the generator fixed. The mean square error (MSE) between the recovered video $\mathcal{G}(\mathcal{E}(E))$ and the input data $I$ is defined as the loss function denoted by ${\mathcal{L}}_{\mathcal{E}}$, i.e., $${\mathcal{L}}_{\mathcal{E}}=\mathrm{MSE}[\mathcal{G}(\mathcal{E}(E)),I].$$(8)

By minimizing ${\mathcal{L}}_{\mathcal{E}}$, the encoder learns how to correctly extract the latent vectors with temporal coherence from the 2D snapshot. The training of S2V-AE is finished when the reconstructed video quality stops increasing. Weight decay is employed during the training of the encoder to prevent the weights of the encoder from growing too large [60]. Hyper-parameters to be trained in the encoder are defined through a search over a small grid of candidate values using cross-validation with reconstruction performance measured over a freshly generated batch of data examples.

In the training of both the generator and the encoder, the Adam optimization algorithm [61] was employed with a fixed learning rate, set to ${10}^{-3}$ for the training of the generator, and $3\times {10}^{-4}$ for the training of the encoder. Adam’s ${\beta }_1$ and ${\beta }_2$ parameters were set to 0.9 and 0.999 for the training of the generator and 0.5 and 0.9 for the training of the encoder, respectively. Data loading was set at training time so that both scenes and corresponding snapshots were generated on the fly, yielding a virtually infinite amount of training data. Once the completion of both the generator and the encoder training, the S2V-AE was employed to reconstruct dynamic scenes.

To test the feasibility of S2V-AE, we simulated video reconstruction of flying handwritten digits [62]. Each dynamic scene had a size of $(N_x,N_y,N_t)=(64,64,40)$, which produced the snapshot of (103, 64) in size. Snapshots were generated using the forward model of SMART-COSUP [i.e., Eq. (1)]. Simulation results are summarized in Fig. 3. For the flying digits corresponding to 3, 5, and 7, six representative frames in the ground truth and the reconstructed videos are shown in Figs. 3(a)–3(c), respectively. The reconstructed videos are included in Visualization 1. To quantitatively assess the reconstructed video quality, we analyzed the peak SNR (PSNR) and the structural similarity index measure (SSIM) [63] frame by frame [Figs. 3(d) and 3(e)]. The average PSNR and SSIM of the reconstructed results are 22.9 dB and 0.93, respectively. These results demonstrate that the S2V-AE can accurately reconstruct dynamic scenes from compressively acquired snapshots.

Furthermore, to show that the S2V-AE possesses a more powerful ability in high-quality video reconstruction, we compared its performance to U-net, which is most popularly used in video compressed sensing [38]. In particular, this U-net featured a convolutional encoder–decoder architecture with residual connection and used the same loss function in Ref. [38]. To implement the optimal specifications of this U-net based technique, we used an approximate inverse operator ${\mathbf{\Phi }}^{\mathrm{T}}{(\mathbf{\Phi }{\mathbf{\Phi }}^{\mathrm{T}})}^{-1}$ to alleviate the burden in learning the forward model [38,39]. Particular to SMART-COSUP, we defined $\mathbf{\Phi }=\mathbf{T}{\mathbf{S}}_{\mathbf{o}}\mathbf{C}$. Using the compressively recorded snapshot of the scene (i.e., $E$), the initialized input to the U-net is expressed as ${\widehat{I}}_o={\mathbf{\Phi }}^{\mathrm{T}}{(\mathbf{\Phi }{\mathbf{\Phi }}^{\mathrm{T}})}^{-1}E$, which had the same $(x,y,t)$ dimension as the ground truth. Both the initialized input and its ground truth were used to train the U-net to obtain a good inference ability for new training scenes generated on the fly. To compare the results between U-net and the S2V-AE, we reconstructed the flying digits of 3, 5, and 7 (see Visualization 2). Despite resembling a close trace of these moving digits to their ground truths, the U-net reconstruction failed to recognize the digits’ spatial information in each frame. The limited feature extraction ability (imposed by the large number of frames in these scenes) and the requirement of high temporal coherence (broken by the fast and randomly moving traces of the digits in these scenes) are the two main reasons that attribute to the unsuccessful reconstruction using U-net. In contrast, benefiting from its two-step strategy that incorporates spatiotemporal coherence, the S2V-AE has shown superior performance, manifesting in the sharpness of reconstructed digits, the maintenance of high image quality over a large sequence depth, and the capability of handling randomly moving traces.

The proof-of-concept experiments of SMART-COSUP were conducted by imaging an animation of three bouncing balls, whose starting positions and moving directions were randomly chosen (see Visualization 3). This scene had the size of $(N_x,N_y,N_t)=(256,256,100)$, which produced a snapshot with a size of (355, 256). To improve S2V-AE’s reliability for experimentally captured data, we included various experimental conditions in the forward model to train the S2V-AE. In particular, an experimentally captured mask image was used for the spatial modulation operator. Moreover, with consideration of the noise level in the deployed CMOS camera, Gaussian noise with a standard deviation randomly selected from ${10}^{-1}$ to ${10}^{-4}$ was added into the training data to match the SNRs in acquired snapshots. Finally, distortion in the acquired snapshot was corrected by an established procedure [64,65].

Furthermore, the three centroids in each frame were used as vertices to build a triangle. Figures 4(e) and 4(f) show the time histories of the geometric center of this triangle generated from the results of the three reconstruction methods. The standard deviations in the $x$ and $y$ directions averaged over time were calculated as (25.4 μm, 17.0 μm), (14.8 μm, 14.5 μm), and (8.3 μm, 6.7 μm) for TwIST, PnP-ADMM, and S2V-AE, respectively. These results show that the S2V- AE has delivered superior performance in image quality and measurement accuracy.

To show the broad utility of SMART-COSUP, we applied it to tracking multiple fast-moving particles. In the setup, white microspheres were scattered on a surface that rotated at 6800 revolutions per minute [Fig. 5(a)]. The 640 nm continuous-wave laser was used to illuminate the rotating microspheres at an incident angle of $\sim 50°$. To visualize the beads’ continuous motion while capturing a sufficiently long trace, the scattered light was captured by the SMART-COSUP system at 20 kfps. An objective lens (CF Achro $4\times$, Nikon Corp., Tokyo, Japan) was used as the front optics. Figure 5(b) shows a static image of three microspheres (marked as ${\mathrm{M}}_1\text{−}{\mathrm{M}}_3$) around the rotation center. Figure 5(c) shows a time-integrated image of this dynamic event acquired using the CMOS camera in the SMART-COSUP system at its intrinsic frame rate of 20 fps. Due to the low imaging speed, this time-integrated image cannot discern any spatiotemporal details. In contrast, imaging at 20 kfps, SMART-COSUP captures the trajectory of each microsphere, as shown in Visualization 4. The top image in Fig. 5(d) provides a color-coded overlay of five reconstructed frames (from 0.55 ms to 4.55 ms with a 1 ms interval), which are shown individually in the bottom row of Fig. 5(d). The rotation of ${\mathrm{M}}_1$ and ${\mathrm{M}}_3$ at two different radii [i.e., $r_{{\mathrm{M}}_1}$ and $r_{{\mathrm{M}}_3}$ labeled in Fig. 5(b)] is evident.

To quantitatively analyze these images, we calculated the time histories of $x$ and $y$ positions and the corresponding velocities of these microspheres. ${\mathrm{M}}_2$, sitting at the rotation center, barely changes its position. The time histories of the positions and velocities of ${\mathrm{M}}_1$ and ${\mathrm{M}}_3$ follow sinusoidal functions expressed as $$v_{i(x\text{or}y)}(t)={\mathbf{\omega }}_{\mathrm{F}}{r}_{{\mathrm{M}}_i}\sin ({\mathbf{\omega }}_{\mathrm{F}}t+{\alpha }_{i(x\text{or}y)}).$$(9)

Here, $i=1$ or 3, ${\mathbf{\omega }}_{\mathrm{F}}$ denotes the angular velocity, whose value was preset at 0.71 rad/ms (i.e., 6800 rounds per minute), and $r_{{\mathrm{M}}_i}$ denotes the radius of each microsphere’s rotation trajectory. In this experiment, $r_{{\mathrm{M}}_1}=0.44\mathrm{mm}$ and $r_{{\mathrm{M}}_3}=0.64\mathrm{mm}$. ${\alpha }_{i(x\text{or}y)}$ is the initial phase of the $i$th microsphere in either the $x$ direction or the $y$ direction. Thus, the theoretical linear speeds of ${\mathrm{M}}_1$ and ${\mathrm{M}}_3$ are 0.31 m/s and 0.45 m/s, respectively.

Based on the above analysis, we used single sinusoidal functions to fit the measured velocities. The fitted maximum velocities in the $x$ direction and the $y$ direction are 0.30 m/s and 0.32 m/s for ${\mathrm{M}}_1$, and 0.46 m/s and 0.45 m/s for ${\mathrm{M}}_3$. The fitted angular speeds in the $x$ direction and the $y$ direction are 0.71 rad/ms and 0.70 rad/ms for ${\mathrm{M}}_1$ and 0.71 rad/ms and 0.72 rad/ms for ${\mathrm{M}}_3$. The experimentally measured values have a good agreement with the preset angular speed of the rotating surface.

The S2V-AE offers a new real-time reconstruction paradigm to compressed ultrahigh-speed imaging, as shown in Fig. 2(a). The new architecture of the encoder allows mapping a compressively recorded snapshot into a set of low-dimensional latent vectors. After that, the GAN-trained generator maps such latent vectors into frames of the reconstructed video. Using this scheme, the training procedure is divided into two distinct phases: to train a generative model of static frames and to train an encoding model aiming to sample from the generator. By doing so, unlike direct reconstruction approaches, high quality in frame-wise reconstruction can be ensured by the initially trained generator, while the encoding model needs to learn only how to query coherently across time. This scheme brings in benefits to the reconstructed videos in terms of both quality and flexibility. The encoder in S2V-AE preserves coherence in both space and time. Different from previous works [36,37,39], no artificial segmentation is conducted in the S2V-AE, which avoids generating artifacts due to the loss of spatial coherence. The S2V-AE also explicitly models temporal coherence across frames with the Bi-LSTM. Both innovations ensure artifact-free and high-contrast video reconstruction of sophisticated moving trajectories. Meanwhile, the S2V-AE presents a flexible structure with a higher tolerance for input data. In particular, the generator, used in a PnP setting [66], is independent of the system’s data acquisition, which is important for adaptive compressed sensing applications.

The multiple-discriminator framework implemented in the S2V-AE improves training diversity. While able to generate high-quality, natural-looking samples, generators trained under the framework of the GAN have known drawbacks that have to be accounted for at training time. Namely, mode collapse refers to cases where trained generators can generate only a small fraction of the data support [67]. Standard GAN settings do not account for the diversity of the generated data, but instead, the generator is usually rewarded if its outputs are individually close to the real data instances. As such, a large body of recent literature has tackled the mode collapse using different approaches to improve the diversity of the GAN generators [67,68]. Mode collapse is especially critical in the application we consider here. The generator in the S2V-AE must be able to generate any possible frame, which means being able to output images containing any objects (e.g.,  balls or digits) in any position. To ensure that the generator is sufficiently diverse, the S2V-AE implements the multiple-discriminator framework [69,70]. Moreover, each such discriminator is augmented with a random projection layer at its input. More random views of the data distribution aid the generator in producing results that are approximate to the real data distribution.

The S2V-AE enables the development of SMART-COSUP. This new technique has demonstrated the largest sequence depth (i.e., 100 frames) in existing DNNs-based compressed ultrahigh-speed imaging methods [36–41]. The sequence depth, as a tunable parameter, could certainly exceed 100 frames. In this aspect, the performance of the S2V-AE mainly depends on the encoder [Fig. 2(b)] since it needs to extract the same number of latent vectors as the sequence depth. Although a large sequence depth may bring in training instabilities due to vanishing/exploding gradients, our choice of the Bi-LSTM architecture in the S2V-AE could alleviate gradient-conditioning issues relative to standard recurrent neural networks [71]. Thus, we expect the limit of sequence depth to be up to 1000 frames in the current setup. Moreover, although we only experimentally demonstrated the 20 kfps imaging speed in this work, the S2V-AE could be extended to reconstruct videos with much higher imaging speeds. As shown in Eq. (5), SMART-COSUP’s imaging speed is determined completely by the hardware. Regardless of the imaging speed, the operation of the S2V-AE—reconstruction of a 3D datacube from a 2D snapshot—remains the same. Moreover, considering the link between imaging speeds and SNRs, the successful reconstruction of snapshots with different SNRs during the training procedure, as discussed in Section 5, indicates S2V-AE’s applicability to reconstruct videos with a wide range of imaging speeds. Furthermore, SMART-COSUP replaces the DMD with a printed transmissive mask. Despite being inflexible, the implemented pseudo-random binary pattern has better compatibility with diverse dynamic scenes, improves light throughput and image quality, as well as offers a simpler, more compact system arrangement. Along with its real-time image reconstruction, the SMART-COSUP system is advancing toward real-world applications.

In summary, we have developed the S2V-AE for fast, high-quality video reconstruction from a single compressively acquired snapshot. This new DNN has facilitated the development of the SMART-COSUP system, which has demonstrated single-shot ultrahigh-speed imaging of transient events in both macroscopic and microscopic imaging at up to 20 kfps with a real-time reconstructed video size of $(N_x,N_y,N_t)=(256,256,100)$. This system has been applied to multiple-particle tracking. Despite demonstrated only with the SMART-COSUP system, the S2V-AE could be easily extended to other modalities in compressed temporal imaging [19] and single-shot hyperspectral imaging [72,73]. Moreover, by implementing the variational AE [74], the dependence of the encoder on the sensing matrix could be further reduced. SMART-COSUP’s ability to track multiple fast-moving particles in a wide field may enable new applications on particle imaging velocimetry [75] and flow cytometry [76]. All of these topics are promising research directions in the future.

Acknowledgment. The authors thank Patrick Kilcullen for experimental assistance.