Scattering noises in four kinds of lithium niobate crystals with the same double doping system, which are LiNbO3:Fe:Mn, LiNbO3:Ce:Mn, LiNbO3:Ce:Cu, and LiNbO3:Fe:Cu, are observed and compared experimentally. The results show that nonvolatile holographic recording can effectively suppress scattering noise, which mainly depends on recombination coefficients of both the shallower centers and the deeper centers. The small recombination coefficients of the shallower centers and the large recombination coefficients of the deeper centers benefit the amplification of the signal gratings and the suppression of the noise gratings. In addition, the initial seed scattering also impacts the recorded scattering noise, and the little seed scattering results in low scattering noise. The theoretical simulations are performed for confirmation. Among the four kinds of doubly doped crystals, in LiNbO3:Ce:Cu the performances of nonvolatile recording are the best with low scattering noise and high diffraction efficiency.