Image quality evaluation is an important work to optical system design and manufacture. Generally, modulation transfer function and wavefront aberration are two common evaluation ways. For high resolution optical systems, the traditional one-dimensional modulation transfer function has some shortcomings when evaluating their image quality because it only provide one dimensional spatial frequency information. Based on the Fourier power spectrum density theory of random images, a method to measure the two-dimensional modulation transfer function of optical systems was proposed by setting random images as the target. An algorithm of using two-dimensional modulation transfer function to calculate the wavefront aberration directly was developed through simplifying the formula of optical transfer function. In this part, the Newton-Cotes integral formula was used. The experimental results show that the two-dimensional modulation transfer function is better than the one-dimensional modulation transfer function in revealing the true characteristics of imaging optical systems, and that the wavefront aberration calculated with the algorithm have the same contour with the theoretical wavefront aberration. It can be a potential way to measure wavefront aberration of optical systems.