Due to the image blur caused by defocus and motion, a new method is presented to distinguish defocus blur and motion blur images robustly and accurately. Based on Hough transform and compared the number of highlights in the Hough matrix, this method has high accuracy, up to 100%. An improved approach for motion blur direction from motion blur image via two directional derivations is proposed which improves the precision estimation. And, an improved method is presented to calculate the edge spread function using improved Prewiit operator and Fermi function, then obtain the MTF and then restore image using Wiener filter. Some experiments are performed to validate the performance of the proposed method. Experimental results show that this method has effectiveness and strong resistance to the noise, and can work on noisy images with low SNR; when SNR is 20 dB, it can work robustly. And compared with the traditional image restoration method and model, the image restoration results are improved significantly using the proposed method.