Beam quality improvements by a large margin for signal and idler beams of a high energy 100 Hz KTiOAsO4 (KTA) non-critical phase matching (NCPM) optical parametric oscillator (OPO) were demonstrated using an unstable resonator configuration instead of a plane-parallel one. Theoretically, influences of cavity lengths and transmission of an output coupler on the OPO conversion efficiency for both were numerically simulated. For OPO based on an unstable resonator with a Gaussian reflectivity mirror, the maximum pulse energies at the signal (1.53 µm) and idler (3.47 µm) were about 75 mJ and 26 mJ, respectively. The corresponding beam quality factors of the signal were M_x² = 9.8 and M_y² = 9.9, and M_x² = 11.2 and M_y² = 11.5 for the idler. As a comparison, 128 mJ of signal and 48 mJ of idler were obtained with the plane-parallel resonator, and the M² factors of the signal were M_x² = 39.8 and M_y² = 38.4, and M_x² = 32.1 and M_y² = 31.4 for the idler. Compared with a plane-parallel cavity, over eight times and three times brightness improvements were realized for the signal and idler light, respectively.