Object information is seriously degraded after being modulated by complex media [1,2]. The scattering of light in the diffusion is an established problem considered to be a common phenomenon in our daily life (i.e., seeing through dense fog to obtain the license plate and driver’s facial information is crucial for a traffic monitor). Imaging with randomly scattered light is a challenging problem with an urgent requirement in different fields (e.g.,  astronomical observations through the turbulent atmosphere and biological analysis through active tissue) [3–7]. Conventional imaging methods based on geometric optics cannot work with the disordered light beam under scattering. Benefitting from the great progress of optoelectronic devices and computational techniques, many new imaging methods have been proposed for imaging through scattering media. The typical imaging techniques include the wavefront-shaping methods [8–11], reconstruction using the transmission matrix [12,13], single-pixel imaging methods [14–16], or techniques based on the point spread function (PSF) [17–19]. The methods listed here have made great progress in object reconstruction with invasive prior, and the scattering scenes are relatively stable. Speckle correlation based on the optical memory effect (OME) is an extraordinary method for noninvasive imaging through opaque layers [20] with only one frame of speckle pattern [21,22]. Object recovery based on speckle correlation methods uses phase retrieval algorithms such as hybrid input-output (HIO) [23], those based on the alternating direction method of multipliers (ADMM) [24], and phase retrieval based on generalized approximate message passing (prGAMP) [25]. The field of view (FOV) of speckle-correlation methods is limited by OME, and the recovery performance is also influenced by the recovery capability of phase retrieval algorithms.

Recently, with the advent of digital technology, big data, and advanced optoelectronic technology, deep learning (DL) has shown great potential in optics and photonics [26,27]. With powerful data mining and mapping capabilities, the data-informed DL methods can extract the key features and build a reliable model in many fields [28]. To date, the DL approach has been successfully applied in digital holography imaging [29–32], Fourier ptychographic imaging [33–36], computational ghost imaging [37,38], superresolution microscopic imaging [39–42], optical tomography imaging [43–45], photon-limited imaging [46,47], three-dimensional (3D) measurements with fringe pattern analysis [48–51], and imaging through scattering media [52–60]. Compared to traditional computational imaging (CI) technology, the learning-heuristic methods can not only solve complex imaging problems, but also can significantly improve the core performance indicators (i.e., spatial resolution, temporal resolution, and sensitivity). The great progress by DL is indicated by the rapidly increasing number of DL-related publications in photonics journal in the last several years [61]. However, the methods using DL are meeting several challenging problems, such as the choice of the DL framework tending to be empirical and the limited generalization capability.

Based on the nonlinear characteristics of deep neural networks (DNNs), DL methods have good performance in highly ill-posed problems, especially in imaging through random media [52–54,57]. IDiffNet is the first proposed method to reconstruct an object through scattering media via a densely connected DNN. The performance with a different type of training dataset and the loss function are systematically discussed [52]. A hybrid neural network is constructed to see through a thick scattering medium and achieves object restoration exceeding the FOV of OME [53]. The speckle patterns of single-mode fiber and multimode fibers are reconstructed and recognized successfully [54]. PDSNet is built to reconstruct complex objects through scattering medium and expands the FOV up to 40 times of the memory effect (ME) [55]. The methods above are mainly focused on a specified diffuser that has a limitation in complex and variable scattering conditions. Therefore, some DL methods have the potential to reconstruct objects through unstable media that mainly use different DNN structures, such as one-to-all with dense blocks, interpretable DL method, generative adversarial network (GAN), or a two-stage framework [57–60]. Li et al. [57] first proposed a DL technique to generalize from four different diffusers to more diffusers with raw speckles, which requires the unknown diffusers to have similar statistical properties and the structure of objects to be simple [57]. Almost all the DL methods for imaging through scattering media use speckle patterns directly, and more information might be further excavated with a traditional physical theory. The efficient physics prior can provide an optimized direction for DNN to find the optimal reconstruction solution in different scattering scenes. After being modulated by different diffusers, the scattering light with photons walk randomly brings about the great statistical difference of speckle patterns even with the same one object. Although it has been proven that the DL method, which focuses on DNN structure design, has the generalization capability to reconstruct hidden objects through unknown diffusers, it is still difficult to obtain an accurate object structure under the condition of fewer training diffusers and has a limitation in reconstructing the complex object [57]. At the same time, the generalized diffusers should have similar statistical characteristics. Therefore, in the absence of effective physical constraints and guidance, DL methods can hardly extract universal information from speckle patterns under highly degenerating conditions. Solely data-driven DL methods will lead to the limited generalization capability that the model is over-relying on training data. Thus, to solve the problems of imaging through multicomplex media, combining the scattering theory with DNN is a more efficient method than designing specific DNN structures.

In this paper, with the physics prior of scattering and the support of DL, a physics-informed learning method is proposed for imaging through unknown diffusers. By pre-processing, the data model based on the physics prior can solve the generalization problems in different scattering scenes, which can reduce the data dependence of the DL model and robustly improve the feature extraction efficiency. The efficient physics prior can provide an optimized direction for DNN to find the optimal reconstruction solution in different scattering scenes. The DL method based on physics prior can help to learn and extract the statistical invariants from different scattering scenes. Instead of training with captured patterns directly, using the DL framework with speckle correlation prior to imaging through different diffusers is technologically reasonable. Employing the physics-informed learning method, scalable imaging through unknown diffusers can be achieved with high reconstruction quality. The scattering degradation of the sparse objects can even be modeled with one ground glass, and imaging through unknown ground glasses even with different statistical characteristics can be achieved. More complex objects (e.g.,  human faces) can be reconstructed accurately by slightly increasing the number of training diffusers. Meanwhile, it is hard to restore the objects efficiently exceeding the FOV of OME by the traditional speckle-correlation method. Based on the powerful capability in data mining and processing of DNN, the physics-informed learning method can also break through the FOV limitation for scalable imaging. Finally, we demonstrate the physics-informed learning scheme with an experimental dataset and present the quantitative evaluation results with multiple indicators. The results with the statistical average indicator show the accuracy and robustness of our scheme, and reflect the great potential of combining physical knowledge and DL.

The proposed model must be established to have general applicability for scalable imaging through unknown diffusers, and it is also one of the indispensable conditions to apply this method to practical complex scenes. The wave propagating through an inhomogeneous medium with multiple scattering will generate a fluctuating intensity pattern, and the universal physical law exists in different transmitted modes. The speckle correlation and memory effect in optical wave transmission through disordered media are proposed to observe and analyze the shift-invariant characteristic of speckle patterns [62,63]. The speckle patterns of scattered light through diffusive media are invariant to small tilts or shifts in the incident wavefront of light, and the outgoing light field still retains the information carried by the incoming beam within the range of ME [64]. Therefore, within the scope of ME, the scattering system can be considered as an imaging system with shift-invariant point spread function. The speckle pattern captured by the camera is given by the convolution of the object intensity pattern $O(x)$ with the PSF ($S$), which can be calculated by $$I=O\ast S,$$(1)where the symbol $\ast$ denotes the convolution operator. Using the convolution theorem, the autocorrelation of camera pattern intensity can be defined as $$I\star I=(O\ast S)\star (O\ast S)=(O\star O)\ast (S\star S),$$(2)where the $\star$ is the correlation operator and $S\star S$ is a sharply peaked function representing the autocorrelation of broadband noise. The autocorrelation of the speckle pattern is approximately equal to the autocorrelation of the object hidden behind the scattering media, and the speckle autocorrelation has an additional constant background term C [21]. Thus, Eq. (2) can be further simplified as $$I\star I=(O\star O)+C.$$(3)When the object size exceeds the range of OME, the object can be divided into multiple objects $O_i$ within the OME scope and $n$ represents the object distributed in $n$ different OME ranges (see Appendix A for details). Thus, the autocorrelation distribution of the speckle pattern exceeding OME can be defined as $$I\star I=\sum _{i=1}^n(O_i\star O_i)+C^{\prime }.$$(4)

With the speckle-correlation prior, the statistical invariants of the object through different scattering media can be effectively extracted, which informs the DNN to obtain useful information and reconstruct the object in different scattering scenes. Imaging through different scattering media with speckle-correlation prior can be used as a reference and a heuristic approach to design the DL methods in different optical problems.

After the speckle-correlation pre-processing step, the captured speckle pattern is adjusted and refactored, and the next step is post-processing by DNN to reconstruct the hidden object. By adding the speckle-correlation theory, the imaging model can make full use of the advantages of the neural network. The DL model is a simple convolutional neural network (CNN) of the U-Net type [65]. Comparing a specially designed DNN structure, the physics-informed learning method can achieve better imaging results with a simple U-Net without any other tricks.

In our experiments, multiple objects datasets with different levels of complexity are used to reconstruct through different diffusers, such as the modified National Institute of Standards and Technology (MINIST) dataset [66] and FEI face dataset [67]. An equilibrium constraint loss function is important for the training process, and we design a combination loss that includes negative Pearson correlation coefficient (NPCC) loss and mean square error (MSE). The Pearson correlation coefficient is an index used to evaluate the similarity between two variables, and the calculated value is distributed from $-1$ to 1. A negative value represents a negative correlation, a positive value represents a positive correlation, and 0 represents an irrelevant correlation. Since the optimization direction of the deep learning is optimized in the direction of loss value reduction, to obtain a positive reconstruction result, the NPCC is used for training [52]. The loss functions can be formulated as $$\text{Loss}={\text{Loss}}_{\mathrm{NPCC}}+{\text{Loss}}_{\mathrm{MSE}},$$(6)$${\text{Loss}}_{\mathrm{NPCC}}=\frac{-1\times \sum _{x=1}^w\sum _{y=1}^h[i(x,y)]-\widehat{i}][I(x,y)]-\widehat{I}]}{\sqrt{\sum _{x=1}^w\sum _{y=1}^h[i(x,y)]-\widehat{i}{]}^2}\sqrt{\sum _{x=1}^w\sum _{y=1}^h[I(x,y)]-\widehat{I}{]}^2}},$$(7)$${\mathrm{Loss}}_{\mathrm{MSE}}={\mathrm{Loss}}_I=\sum _{x=1}^w\sum _{y=1}^h{|\tilde{i}(x,y)-I(x,y)|}^2,$$(8)where $\widehat{I}$ and $\widehat{i}$ are the mean value of the object ground truth $I$ and the DNN output $i$, respectively, and $\tilde{i}$ is a normalized image of $i$. The combination loss function has a good capability to reconstruct objects with different complexity and sparsity through different scattering media. To train the DNN, an Adam optimizer is selected as the strategy to update the weights in the training process. The DNN is performed on PyTorch 1.4.0 with a Titan RTX graphics unit and i9-9940X CPU under Ubuntu 16.04.

To obtain the speckle patterns in different scattering scenes, nine different ground glasses are used as the diffusers in the experiments, including six 220 grit diffusers (D1–D6), one 120 grit diffuser (D7), one 600 grit diffuser (D8) produced by Thorlabs, and one 220 grit diffuser (D9) produced by Edmund, like the configuration in Section 2.B. We choose one ground glass (D1) or the first three pieces of ground glasses (D1, D2, and D3) as the training diffusers, and the remaining ground glasses as the test diffusers. The objects are mainly selected from the MINIST database and FEI face databases. The character objects are selected randomly from the MINIST dataset to form the different complexity of single-character and double-character objects. For collecting the experimental data, 600 single characters, 600 double characters, and 400 human faces are used as objects hidden behind each diffuser. The first 500 characters are used as the seen objects and the remaining characters are used as the unseen objects. Similarly, the first 360 human faces are used as seen objects and the remaining faces are used as unseen objects. The autocorrelation pre-processing for speckle patterns is the first step for our method. As for the processing of the speckle patterns, we take the $512\times 512$ camera pixels from the center pattern to calculate the autocorrelation and crop the center to $256\times 256$ pixels autocorrelation pattern as the input autocorrelation image. All the objects, speckle patterns, and autocorrelation images are in grayscale in this experiment.

According to different training data and testing data, different groups are used to characterize the generalization capability of the physics-informed DL method, respectively. All of the testing data are captured from unknown diffusers for emphasizing the generalization. The data can be roughly divided into four groups.

Group 1: The objects are the single characters within the OME. The training data can be divided into two types: training with one diffuser (D1) or three diffusers (D1–D3) with seen objects (the first 500 characters). The testing data can also be divided into two types: the seen objects and the unseen objects (the last 100 characters) with testing diffusers (D4–D9).

Group 2: The objects are the double characters within the OME. The data arrangement is similar to Group 1, except for the complexity of objects.

Group 3: The objects are the human faces within the OME. The training data can also be divided into two types: training with one diffuser (D1) or three diffusers (D1–D3) with seen objects (the first 360 faces). The testing data can also be divided into two types: the seen objects and the unseen objects (the last 40 faces) with testing diffusers (D4–D9).

Group 4: The objects are the single characters extending the FOV to 1.2 times. The data arrangement is also similar to Group 1, except for the size and distribution of objects.

According to the experimental results shown in Section 3, we have three key points. (i)A physics-informed DL framework is proposed for scalable imaging through different scattering scenes in which the diffusers are previously untrained. The objects hidden behind the unknown diffusers are not limited to simple sparse characters, and more complex objects (e.g.,  human faces) can be reconstructed with high accuracy. The physics-informed learning method can also extend the FOV of conventional speckle-correlation methods.(ii)The DL framework has a reliable generalization capability in imaging through unknown thin scattering media using only one training diffuser for the sparse object. With the number of training diffusers increasing, the generalization capability of the methods is further improved. The proposed method can still reconstruct the overall structure and local details for human faces, even the slight micro-expressions can be clearly distinguished. However, the DL models are prone to preferentially fit the category of the training dataset, which limits the generalization capability of the physics-informed learning method with unknown category objects.(iii)Benefitting from the great capability in data-mining and mapping of DNN, reliable generalization results can also be obtained through unknown diffusers with the extended FOV. Meanwhile, the FOV of the physics-informed learning method is also relevant to several factors, such as the number of training diffusers and the complexity of the hidden objects.

In this paper, a physics-informed learning method is introduced to imaging through diffusers. Specifically, an explicit framework is established to efficiently solve the generalization problems in different scattering scenes by combining the physics theories and DL methods. This is a new approach to solve scalable imaging with deep learning, which can reconstruct complex objects through different scattering media, and provide an expanded FOV for the real imaging scenes. In the future, more complex scenes and objects can be considered, which can be applied to volumetric multiple scattering, such as biological imaging and astronomical imaging.

Acknowledgment. The authors thank Qianying Cui, Yingjie Shi, Chenyin Zhou, Kaixuan Bai, and Mengzhang Liu for technical support and experimental discussions.