It is hard and costly to test resolution directly, because the focal length of testing equipment could be nearly ten meters. Solving resolution by diffraction wavefront aberration indirectly is an effective solution to this problem. A normalization model of solving resolution using fitting coefficients of Zernike polynomials was established based on the spectral imaging theory of Fourier optics. The relationship between resolution and wavefront aberration of diffraction gratings was illustrated by this model. Finally, a new method of testing resolution using fitting coefficients of Zernike polynomials was proposed. According to this method, the resolution of a grating is tested by ZYGO interferometer indirectly. Compared with direct method, results indicate that the error of indirect method is less than 4.22%, and this method could be an effective way to avoid the difficulty of direct method to solve resolution. Meanwhile, this method can be used in ZYGO interferometer to solve resolution by wavefront testing easily.