Aiming at the problem of high reflection and poor low-frequency radiation performance of TEM horn antenna, the simulation optimization design is carried out. The low frequency compensation methods of end loading resistance and back loading resistance are analyzed by using time domain analysis method, and the structural parameters such as the size of the TEM horn antenna and the apex angle of the plate are optimized. The peak field strength and pulse width of the radiation field are improved by using the end crimping structure. According to the simulation results, TEM horn antenna with length of 2.5 m, the included angle of the plates of 45° and antenna plate apex angle of 45° is developed. The simulation results are verified by the measurement of the new antenna. The results show that the loading resistance can effectively reduce the reflection, and the peak field strength and pulse width of the back loading resistance are higher than that of the end loading resistance, and the effect of 4 resistors in parallel loading is better. The radiation performance of the antenna can be improved by appropriately increasing the antenna length, the apex angle of the plate and the included angle between two plates. The research results provide a reference for the application of TEM horn antenna in the radiation nuclear electromagnetic pulse test system.