Beam quality improvements by a big margin for signal and idler beam of a high-energy 100-Hz KTiOAsO4 (KTA) optical parametric oscillator were demonstrated using unstable resonator configuration instead of plane-parallel one. Theoretically, influences of cavity lengths and transmission of output coupler on the OPO conversion efficiency for both were numerically simulated. For OPO based on unstable resonator with a Gaussian reflectivity mirror (GRM), the maximum pulse energies at signal (1.53 μm) and idler (3.47 μm) were about 75 mJ and 26 mJ, respectively. The corresponding beam quality factors of signal were M2 x=9.8 and M2 y=9.9, and M2 x=11.2 and M2 y=11.5 for the idler. As a comparison, 128 mJ of signal and 48 mJ of idler were obtained with the plane-parallel resonator, and the M2 factors of the signal were M2 x=39.8 and M2 y=38.4, and M2 x=32.1 and M2 y=31.4 for the idler. Compared with a plane-parallel cavity, over 8 times and 3 times brightness improvements were realized for the signal and idler light, respectively.