Ultrafast imaging has played an indispensable role in photochemistry [1,2], biomedicine [3–5], microfluidics [6], shock waves [7], and plasma physics [8]. Recently, various ultrafast imaging techniques have been developed, including compressed ultrafast photography (CUP) [9–11]. Unlike some active ultrafast imaging techniques that need specific illumination light [12–14] or a pump–probe technique that requires multiple measurements [15–17], CUP is a single-shot and passive ultrafast imaging technique. Its temporal resolution and number of frames can reach tens of femtoseconds and several hundred, respectively. Therefore, CUP has great advantages for measuring some self-luminous or non-repeatable ultrafast phenomena, which is attributed mainly to the novel model of CUP, which combines compressed sensing (CS) theory and time–space conversion technology. So far, CUP has been successfully applied to measure light reflection and refraction [9], femtosecond temporal focusing [10], photonic Mach cones [18], dissipative solitons [19], phase-sensitive transparent objects [20], three-dimensional (3D) objects [21], ultrashort laser spatiotemporal evolution [22], and photoluminescence processes [9]. However, due to the high data compression ratio, the fidelity of reconstructed images for CUP is relatively low by the conventional two-step iterative shrinkage/thresholding (TwIST) algorithm, which limits its practicality. To improve image fidelity, a variety of methods have been proposed, such as a space- and intensity-constrained image reconstruction algorithm [23], augmented-Lagrangian (AL)-based image reconstruction algorithm [24], plug-and-play alternating direction method of multipliers algorithm [25], optimizing the codes for CUP [26], lossless CUP [18], and multi-encoding CUP [27]. These proposed schemes can improve image fidelity to a certain extent, but there are still great challenges in measuring the complex dynamic scenes.

In image reconstruction of CUP, all selections of the sparse domain, determination of relevant iteration parameters, and denoising after iteration calculation greatly limit image fidelity. To completely solve these problems, we developed a novel image reconstruction method based on an AL and deep-learning (DL) hybrid (i.e., AL+DL) algorithm. This idea is borrowed mainly from some early algorithms, such as the AL algorithm [24,28,29], learning iteration parameters [30–33], learning sparse domain [34–37], and U-net architecture [38], but there are still many differences compared to each of the early algorithms. First, the AL+DL algorithm utilizes multiple learning transformations to seek the best sparse domain. Typically, the sparse domain in conventional TwIST and AL algorithms is determined before image reconstruction [24,39], so it is usually not optimal for one dynamic scene. In contrast, the sparse domain in the AL+DL algorithm can be optimized in multiple transformations, which is more pertinent. Second, the AL+DL algorithm takes full advantage of gradient descent (GD), DL, and AL algorithms, which simplifies the mathematical architecture to deal with the 3D tensor problem, and these advantages can reduce the cost of each iteration and decrease the number of iterations. Third, the AL+DL algorithm optimizes the relevant iteration parameters by learning the dataset, which is different from previous AL and TwIST algorithms, where these parameters are artificially predetermined. Finally, the AL+DL algorithm uses a U-net architecture containing attention layers to help denoise and retain the spatial details of the images after iteration calculation. Importantly, our theoretical simulation and experimental results show the AL+DL algorithm can obtain much higher image fidelity than conventional AL and TwIST algorithms for CUP, which strongly supports our theory.

In CUP, a 3D dynamic scene I(x,y,t) is encoded by operator C, sheared by operator S, and integrated by operator T, and finally a two-dimensional (2D) image E(x′,y′) is obtained. For convenience, hereafter, I(x,y,t) is abbreviated to I, and E(x′,y′) is abbreviated to E. Mathematically, this process can be described as E=TSCI.(1)

For simplicity, we define O=TSC. Thus, Eq. (1) can be further written as E=OI.(2)

To recover 3D I from 2D E, we need to solve the inverse problem of Eq. (2). The number of elements in I is much larger than that in E, so the inverse problem of Eq. (2) is undetermined. The CUP strategy is to introduce a CS theory [9]. The CS theory makes full use of the sparsity of I in a certain domain to recover the original information. This sparsity in one domain means that only a few elements are nonzero, while most of the elements are zero. Consider a case in which I has n elements and E has m elements in the original domain, and I has s nonzero elements in a sparse domain, i.e., the sparsity where n≫s and n>m>s. Due to the fact that m is generally larger than s, this makes it possible to solve the inverse problem of Eq. (2). In a practical solution, the CS algorithm minimizes I in a sparse domain on condition of Eq. (1), which is shown as {minIΦ(I)s.t.E−OI=0,(3)where Φ(I) is the expression of I in the sparse domain. According to CS theory [40,41], the original dynamic scene I can be completely recovered when m>fsμ2,(4)where f is a constant correlated with the number of elements n, μ is the mutual coherence between the sparse basis of the sparse domain Φ and operator O, determined by operators C, S, and T. From Eq. (4), one can see that both increasing m and reducing s and μ are feasible schemes to improve the quality of image reconstruction. However, increasing m, i.e., increasing the sampling rate, will reduce the spatial resolution or requires many streak cameras, which is impractical in the actual CUP system. Thus, reducing s and μ is the best choice. Optimizing O can reduce only μ, while optimizing Φ can reduce both s and μ; therefore, here we employ the method of optimizing Φ. To optimize Φ, we impose a low-rank property on the entire dynamic scene I (tensor) with many different transformations [42], which is different from traditional methods with only one transformation. Thus, problem (3) can be further written as {minI(x,y,t)∑pqψpIs.t.E−OI=0,(5)where ψp represents one transformation in the sparse domain, and q denotes the total number of transformations. In transforming problem (5) from a constraint into an unconstraint, there exist two frameworks: the penalty function method and the AL method. The performance of the AL method is better than that of the penalty function method, which has been proved in previous works [24,29], and therefore, here the AL method is adopted. Thus, problem (5) can be transformed into minI{∑pqψpI−γ(E−OI)+ζ2‖E−OI‖F2},(6)where γ and ζ are the Lagrangian multiplier and penalty parameter, respectively, which are associated with I. For convenience, problem (6) is further written as minI{∑pqψpI+ζ2‖E−γζ−OI‖F2}.(7)

To solve problem (7), an auxiliary variable J is introduced into problem (7), and is written as {minI{∑pqψpJ+ζ2‖E−γζ−OI‖F2}s.t.I=J.(8)

By adopting the AL method, the constrained problem (8) can be transformed into minI,J{∑pq[ψpJ−λp(I−J)+δp2‖I−J‖F2]+ζ2‖E−γζ−OI‖F2},(9)where λ={λ1,λ2,…,λq} and δ={δ1,δ2,…,δq} are the Lagrangian multipliers and penalty parameters, respectively, which are associated with J in different transformations. By transformation, problem (9) can be further written as minI,J{∑pq(ψpJ+δp2‖I−λpδp−J‖F2)+ζ2‖E−γζ−OI‖F2}.(10)

Problem (10) can be solved by an alternating direction method of multipliers (ADMM) based on an iteration of solving the I-subproblem and J-subproblem alternatively. However, in the J-subproblem, the sparse domains in different transformations lead to different solutions at the beginning of the iteration. Therefore, some independent auxiliary variables W={W1,W2,…,Wq} are introduced for each transformation, and thus problem (10) can be written as minI,W{∑pq(ψpWp+δp2‖I−λpδp−Wp‖F2)+ζ2‖E−γζ−OI‖F2}.(11)

To solve problem (11), the ADMM is also adopted to solve the I-subproblem and W-subproblem alternatively. In the kth iteration, the I-subproblem can be written as Ik=argminI{∑pqδpk2‖I−λpkδpk−Wpk−1‖F2+ζk2‖E−γkζk−OI‖F2},(12)and the W-subproblem can be written as Wpk=argminWpψpWp+δpk2‖Ik−λpkδpk−Wp‖F2.(13)

The I-subproblem in Eq. (12) is a quadratic regularized least-squares problem, and its direct solution is given in a closed form as {Ik=HARD[ζkOTE−γkOT+(∑pfδpkWpk−1−λpk)],HARD=(ζkOTO+∑pfδpkIiden)−1,(14)where Iiden is an identity operator. It is expensive to compute HARD due to large data, so here the GD algorithm is used to solve the I-subproblem. The main shortcoming of the GD algorithm is that the number of iterations is very large because it is difficult to choose the step size. Some methods have been developed to obtain a better step size through much computation, such as the Barzilai and Borwein (BB) method [43,44]. Here, a learning method is used to seek the optimized step size. In our method, the number of iterations is much less than that of the BB method. Based on the GD algorithm, the solution to Eq. (12) can be expressed as Ik=Ik-1−αk[∑pqδpk(Ik−1−λpkδpk−Wpk−1)−ζkOT(E−γkζk−OIk−1)],(15)where α is the step size in the GD algorithm, and here the GD algorithm is utilized to calculate Eq. (15). From Eqs. (15) and (13), one can see that the solution to problem (6) depends mainly on Wp instead of I. For convenience, the solution to Wp can be further written as Wpk=Sp(Ik−λpkδpk).(16)

Here, only three representative pictures are selected, and an interesting area in each dynamic scene is enlarged for observation. Spatial details in the boatman, ocean animal, and finger can be clearly observed by the AL+DL algorithm, while these details are submerged by the AL and TwIST algorithms, which is disadvantageous for high-spatial-resolution imaging of a dynamic scene. To intuitively compare the improved efficiency in image fidelity by the AL+DL algorithm, we calculate PSNR and SSIM, and the calculated results are given in Table 1. Compared to the AL and TwIST algorithms, both PSNR and SSIM by the AL+DL algorithm are significantly improved. Here, PSNR (SSIM) is increased by at least 4.35 dB (0.136) for the boatman, 5.47 dB (0.114) for the ocean animal, and 9.78 dB (0.051) for the finger. Based on these calculated results, a rule can be found, which is, the simpler the spatial structure of the dynamic scene, the higher the improvement efficiency of PSNR, while the improvement efficiency of SSIM shows the opposite behavior. This phenomenon should be related to the sparsity of the dynamic scene; the simpler dynamic scene usually has higher sparsity, and vice versa. PSNR is based on a logarithmic function, which is not very well matched to perceived visual quality, but SSIM is based on visible structures in the image. Thus, PSNR has high improvement efficiency for a simple dynamic scene (i.e., finger), while SSIM has high improvement efficiency for a complex dynamic scene (i.e., boatman). In addition, the AL+DL algorithm can reconstruct a dynamic scene in only a few seconds, which is much shorter than the AL and TwIST algorithms, which need tens of seconds; the computing efficiency is improved by an order of magnitude, which is very beneficial in practical applications of CUP.Table 1.

Average PSNR (in dB) and SSIM by Different Image Reconstruction Algorithms in Different Dynamic Scenes

Here, the static image is achieved by external CCD measurement without encoding operator C and shearing operator S, as shown in Fig. 5(e). Meanwhile, the intensities of Figs. 5(e)–5(h) are also integrated along the horizontal direction, and the calculated results are given on the right of the relative images. The AL+DL algorithm can retain very high image fidelity, but the AL and TwIST algorithms cause a certain degree of image distortion. The fundamental reason should be the mismatch of the sparse domain in image reconstruction. More importantly, like the static image, the blocked part in the laser spot (see light blue squares) can be clearly distinguished by the AL+DL algorithm, where an obvious valley in the intensity curve is observed, but not by either the AL or TwIST algorithm, especially the TwIST algorithm.

The AL+DL algorithm is a data-driven method, which can optimize the sparse domain and relevant iteration parameters by learning instead of hand-crafted determination. For CS, the sparse domain is the core part that determines the sparsity and affects mainly the coherence. Thus, the sparse domain almost determines the image reconstruction quality. In general, the learning method can seek better sparse domain and iteration parameters, and therefore the AL+DL algorithm can get higher image fidelity than conventional AL and TwIST algorithms. Because of learning the sparse domain and iteration parameters, the AL+DL algorithm has high robustness and allows the encoding operator C to be different in training and testing processes, while the pure neural network algorithms cannot, such as deep fully connected networks [58], ReconNet [59], DR²-Net [60], λ-net [38], and DeepCubeNet [61]. Also, the AL+DL algorithm embeds a GD algorithm into tensor computation, which involves massive data. In calculation, the GD algorithm does not easily find the appropriate step size, so it needs to perform many iterations, i.e., the convergence speed is low. To decrease the number of iterations, data scientists prefer Newton’s method or a conjugate gradient algorithm by increasing the cost of each iteration [62]. However, some mathematicians seek a better step size in the GD algorithm to decrease the number of iterations by increasing the cost of each iteration, such as the BB method. Here, we utilize the GD algorithm to calculate the large data by a data-driven method based on the learning model, which can find the optimal step size to decrease the number of iterations without increasing the cost of each iteration and make the gradient show better orthogonality. It is noted that the AL+DL algorithm needs just 15 iterations, while the corresponding traditional algorithm based on the BB method needs more than 100 iterations.

As shown in Figs. 3, 5, and 6, compared to the AL and TwIST algorithms, the AL+DL algorithm shows great advantages in image reconstruction accuracy, but it also inherits the shortcoming of the data-driven method, i.e., the dependence on a learning dataset. In image reconstruction, these images in the dataset should have some similarities to those in the dynamic scene. An inappropriate training dataset may lead to results worse than those obtained by the AL and TwIST algorithms. In some special dynamic scenes, it may be difficult to find a similar dataset for training. In this case, it is feasible to increase the sampling rate m, such as lossless-CUP or multi-encoding CUP. Moreover, it is also a good idea to optimize the codes, which is similar to optimizing the sparse domain, which can reduce coherence. However, the AL+DL algorithm cannot be adopted directly to optimize the codes, because here the codes are considered as constant. Optimizing the codes demands that the mathematical architecture regard the codes as a variable; thus, the whole AL+DL architecture needs to be redesigned. In the future, we will strive to seek some new algorithms to simultaneously optimize the codes, sparse domain, and iteration parameters by learning the dataset.

In summary, we have developed a new AL+DL algorithm to realize high-fidelity image reconstruction for CUP. In our method, there are four key points: (1) optimizing the sparse domain in multiple transformation; (2) optimizing the relevant calculation parameters in the iteration process; (3) employing the GD algorithm to improve computing efficiency; (4) embedding the U-net architecture to help denoise. Key points (1), (2), and (4) are implemented by the DL method, and improving key point (3) also needs the DL method. However, the whole framework is determined by the AL method, which combines these four key points. Thus, the AL+DL algorithm not only utilizes the training neural networks, but also has some potential mathematical interpretations. More importantly, these results from theoretical simulations and experimental measurements show that the AL+DL algorithm is superior to conventional AL and TwIST algorithms in image fidelity and computing efficiency. Additionally, the AL+DL algorithm is a simple mathematical architecture, so it is easy to extend to other high-dimensional tensor fields. In future studies, we will continue to search for better image reconstruction algorithms for CUP to achieve super-high image fidelity.