Based on the anisotropic diffusion equation, the disadvantages of high computational complexity and unstability in speckle noise elimination with the existing diffusion algorithms are analyzed. A new algorithm of diffusion direction estimation is proposed. The variances of stripe regions in four directions are adopted as the direction measure of local area in the image. The general and diagonal integral images are adopted for fast computation of the variances. The diffusion quantity of each pixel is determined by the differences between the current point and the eight adjacent pixels together with the weight coefficients in corresponding directions. Meanwhile, the diffusion equation and realization process are presented. Experimental results demonstrate that this method has strong adaptability. It can protect the texture details well during noise elimination, and reduce the consumption of time due to few iteration number.