Fig. 1. Physics imaging model meets data-driven diffusion prior. (a) Diffusion models are novel generative models that produce images progressively in $T$ steps, following the gradient of learned image distributions. Although they produce high-quality images, they are often stuck in local optima and deviate from the true image observed by SPI. (b) Our APD-Net uses the forward model as a measurement consistency constraint to iteratively guide the generation process. The alternative supervision from dual-priors in harmony can significantly improve the low-sampling performance of SPI reconstruction.

Fig. 2. Overall framework of APD-Net. APD-Net integrates two distinct image priors for SPI reconstruction, namely, (a) physics prior and (b) data-driven diffusion prior. The physics imaging prior, defined by specific imaging setups, can be explicitly represented with matrix operations. The diffusion prior, on the other hand, is free from SPI setups and is implicitly learned from numerous high-quality images. During (c) the SPI reconstruction, the intermediate variable ${\mathbf{x}}_{0|t}$ from the diffusion model is iteratively refined with a measurement consistency projection step to inject the physics imaging prior into the reconstruction process for ultra-low-sampling SPI reconstruction.

Fig. 3. Visualization of the model-based physics-guidance process. (a) Target scene. (b) Intermediate image after the diffusion denoising step, which at this point may appear overly blurry or smoothed. (c) Intermediate image after applying the model-based consistency correction step, where the image is refined to reflect more accurate details, although noise and artifacts may remain. Such noises and artifacts can be effectively removed by the following diffusion-denoising steps. (d) Influence from the physics-guidance step, which extracts information from the SPI observation and injects it into the SPI reconstruction process.

Fig. 4. Visual analysis of low-sampling reconstruction and adaptability. (a) Performance of SPI on different sampling rates. (b) Performance of SPI on different modulation patterns. We note that the performance of the previous physics-informed method heavily relies on the accuracy of the pseudo-inverse $A^{†}y$. By contrast, our proposed APD-Net has better performance in all scenarios. Images are taken from Set5 and CelebA datasets.

Fig. 5. Convergence analysis and reconstruction acceleration. (a) Intermediate predictions during the progressive 100-step APD-Net generation process. (b) Plot for PSNR of intermediate predictions. (c) Visual analysis of the final prediction of APD-Net with different skipped steps. (d) Plot for PSNR of predictions with different skipped steps. Images are taken from the CelebA dataset.

Fig. 6. Analysis of noise robustness. (a) Visual comparison with model-based TV and deep learning-based PIDL. (b) Quantitative performance at different noise levels. Images are taken from the Set5 dataset.

Fig. 7. Optical SPI setup for real-world validation. A projector projects extended Bayer-modulated structural light to the target object. A photodetector gathers the light signal reflected from the object and transforms it into an electric signal, which is read out by an oscilloscope.

Fig. 8. Visualization for real-world SPI reconstruction with ultra-low sampling and different modulation patterns. (a) Different sampling rates. (b) Different modulation patterns.

Fig. 9. Visualization for the performance of accelerated reconstruction.

Fig. 10. Visualization for the effectiveness of the extended Bayer color modulation.

Table 1. APD-Net algorithm.

Table 1. Qualitative results on different sampling rates. Both model-based iterative methods (GD, TV) and a deep-learning method (PIDL) with its fine-tuning variant are selected for comparison.