A mathematical model of the optical resonator of high energy laser is set up inthis paper specially for the unstable resonator of high energy laser by using the method ofmatrix transmission optics combining with diffraction integral equation. The results ofcomputer simulation show that the model is correct and some important results areachieved.