Digital holographic system is a promising image-forming system, but speckle noise in the coherent light source of digital holographic system adversely affects the quality of holograms. There are some disadvantages in conventional experimental noise reduction or traditional neural network-based noise reduction methods. In order to realize speckle noise reduction in holograms and balance the efficiency of noise reduction, a fast noise reduction algorithm based on convolutional neural network for single hologram is proposed, and the speckle noise dataset is used to train multilevel neural networks. Theoretical analysis and experimental results show that the convolution neural network applied in digital hologram spectrum domain denoising can effectively improve the quality of the hologram, and multilevel speckle noise can be effectively processed by only one hologram. which can save the effective interference fringes of holograms to the maximum extent while maintaining the denoising performance.