Single-image rain removal aims to remove rain streaks from rainy images and restore rain-free images, to provide support for subsequent tasks such as detection and tracking. However, current rain removal methods have problems, such as image blur, detail loss, and color blur after rain removal. To address the limitations of existing methods, a two-stage rain removal and residual diffusion detail recovery network is proposed. In the first stage, to strengthen local rain streak feature learning and improve global information utilization ability, a dual-attention module (DAB) and a dual-attention U-shaped network (DAU-Net) are proposed by combining channel attention and multi-head self-attention mechanisms, so that the model can dynamically identify various rain streaks and remove them. In the second stage, the characteristics of the diffusion model that first constructed the overall semantic information and captured detailed information are used to leverage the powerful generation ability. The rain removal results of the first stage are used as conditions to guide the diffusion model to reverse sample and generate residual information, to address the problem of detail loss and image blur. Experimental results show that the proposed method performs well on both synthetic and real datasets. On the Rain100H and Rain100L test sets, peak signal-to-noise ratio (PSNR) of 31.75 dB and 39.12 dB and structural similarity (SSIM) of 0.912 and 0.981 are obtained, respectively. The two-stage rain removal network can effectively various complex rain streaks in various scenes and recover more of the image details to achieve better visual effects.