Terahertz (THz) in-line digital holography is a promising full-field, lens-free, and quantitative phase-contrast imaging method with an extremely compact and stable optical configuration. Hence, it is suitable for the application of THz waves. However, the inherent twin-image problem can impair the quality of its reconstructions. In this study, a novel learning-based iterative phase retrieval algorithm, termed as physics-enhanced deep neural network (PhysenNet), is introduced. This method combines a physical model with a convolutional neural network to mitigate the twin-image issue in THz waves. Notably, PhysenNet can reconstruct the complex fields of a sample with high fidelity from just a single in-line digital hologram, without the need for constraints or a pre-training labeled dataset. Based on simulations and experimental results, it is evident that PhysenNet surpasses existing phase retrieval algorithms in imaging quality, further enhancing the application range of THz in-line digital holography.