A fast restoration algorithm based on image quality assessment is proposed to improve the quality of space-variant defocus blurred infrared images. First, the defocus image is restored by the truncated constrained least-squares algorithm with different point spread functions to obtain and perform deringing on multiple restored images. Then, the area centered on each pixel in the restored image is evaluated through an image quality assessment, and the images restored with different parameters are combined according to the image quality assessment to obtain the final restored image. Because there is no need to estimate the point spread function of the blurred image, the truncated constrained least-squares algorithm of spatial calculation is used for restoration. The experimental results show that the algorithm proposed in this paper can quickly restore the space-variant defocused blurred infrared image and is much faster than the method based on point spread function estimation.