The relation between structure of entrance pupil of Golay3 telescope system and F-number of primary mirror is analyzed. Point spread function (PSF) and modulation transfer function (MTF) of telescope system are derived through entrance pupil function which considers the tilt error of one submirror; The effect of tilt error on PSF and MTF is simulated by Matlab program; a example of Golay3 telescope is designed by Zemax software, the PSF and MTF of the system are obtained with tilt error in a submirror, and compared with the theoretical simulation results. Finally, Golay3 telescope′s imaging capability is evaluated under different tilt error value with a standard board as the object. According to the result of the Zemax software, the change of both PSF and MTF of Golay3 telescope system because of tilt error is consistent with that of the theoretical simulation. When the peak-to-valley value of tilt wavefront is smaller than one wavelength, both the main lobe and the side lobes of PSF and MTF pattern vary enormously, the direction of the variation depends on the position of the submirror with tilt error. When the peak-to-valley value of tilt wavefront exceeds one wavelength, both the PSF and MTF remain unchanged basically. The imaging results show that with the increase of tilt error of the submirror in x direction, the maximal recognizable line pairs in y direction changes dramatically.