An error diffusion algorithm based on the angular-spectral diffraction theory is proposed. The complex amplitude hologram of a three-dimensional object is calculated by the layered angular-spectral algorithm. The error diffusion method is used to obtain the phase-only hologram. The clear reconstructed image is reconstructed and the speckle noise of the phase-only holographic reconstructed image for a three-dimensional object is suppressed. The simulation experimental results verify the feasibility and superiority of this algorithm. This proposed algorithm can significantly improve the quality of reconstructed images.