Single-pixel imaging (SPI) enables efficient sensing in challenging conditions. However, the requirement for numerous samplings constrains its practicality. We address the challenge of high-quality SPI reconstruction at ultra-low sampling rates. We develop an alternative optimization with physics and a data-driven diffusion network (APD-Net). It features alternative optimization driven by the learned task-agnostic natural image prior and the task-specific physics prior. During the training stage, APD-Net harnesses the power of diffusion models to capture data-driven statistics of natural signals. In the inference stage, the physics prior is introduced as corrective guidance to ensure consistency between the physics imaging model and the natural image probability distribution. Through alternative optimization, APD-Net reconstructs data-efficient, high-fidelity images that are statistically and physically compliant. To accelerate reconstruction, initializing images with the inverse SPI physical model reduces the need for reconstruction inference from 100 to 30 steps. Through both numerical simulations and real prototype experiments, APD-Net achieves high-quality, full-color reconstructions of complex natural images at a low sampling rate of 1%. In addition, APD-Net’s tuning-free nature ensures robustness across various imaging setups and sampling rates. Our research offers a broadly applicable approach for various applications, including but not limited to medical imaging and industrial inspection.