Diffraction tomography is a promising, quantitative, and nondestructive three-dimensional (3D) imaging method that enables us to obtain the complex refractive index distribution of a sample. The acquisition of the scattered fields under the different illumination angles is a key issue, where the complex scattered fields need to be retrieved. Presently, in order to develop terahertz (THz) diffraction tomography, the advanced acquisition of the scattered fields is desired. In this paper, a THz in-line digital holographic diffraction tomography (THz-IDHDT) is proposed with an extremely compact optical configuration and implemented for the first time, to the best of our knowledge. A learning-based phase retrieval algorithm by combining the physical model and the convolution neural networks, named the physics-enhanced deep neural network (PhysenNet), is applied to reconstruct the THz in-line digital hologram, and obtain the complex amplitude distribution of the sample with high fidelity. The advantages of the PhysenNet are that there is no need for pretraining by using a large set of labeled data, and it can also work for thick samples. Experimentally with a continuous-wave THz laser, the PhysenNet is first demonstrated by using the thin samples and exhibits superiority in terms of imaging quality. More importantly, with regard to the thick samples, PhysenNet still works well, and can offer 2D complex scattered fields for diffraction tomography. Furthermore, the 3D refractive index maps of two types of foam sphere samples are successfully reconstructed by the proposed method. For a single foam sphere, the relative error of the average refractive index value is only 0.17%, compared to the commercial THz time-domain spectroscopy system. This demonstrates the feasibility and high accuracy of the THz-IDHDT, and the idea can be applied to other wavebands as well.