During the course of the image restoration degenerated by the Gauss (G) class Point Spread Function (PSF), an improved image restoration algorithm is proposed based on frequency spectrum characteristic of the blur image. According to the frequency spectrum information, the algorithm improves the course of estimation PSF by utilizing the basic principle of the APEX method. The main direction of the frequency spectrum of the blur image is fitted by the weighted least squares algorithm. The PSF is estimated by using the main direction data of frequency spectrum of the blur image, so more available data will be obtained for the PSF estimation, which reduces the error of the PSF estimation. The method is tested by the simulated blur image and the captured real blur image, and it is evaluated by the subjective vision and the Peak Signal-to-noise Ratio (PSNR). The experiment results indicate that, compared with the method using the frequency-spectrum data of non-main direction data, the improved method ameliorates the restoration effect of the blur image.