With rapid development of sustainable energies and energy conversion technologies, application prospect of thermoelectric (TE) materials in power generation and cooling has received increasing attention. The requirement of improving TE materials with high figure of merit becomes much more important. How to obtain the low lattice thermal conductivity is one of the main concerns in TE materials. In this review, the influences of specific heat, phonon group velocity and relaxation time on the lattice thermal conductivity are discussed, respectively. Several typical features of TE materials with intrinsic low lattice thermal conductivity are introduced, such as strong anharmonicity, weak chemical bonds and complex primitive cells. Introducing multiscale phonon scatterings to reduce the lattice thermal conductivity of known TE materials is also presented and discussed, including but not limited to point defect scattering, dislocation scattering, boundary scattering, resonance scattering and electron-phonon scattering. In addition, some theoretical models of the minimum lattice thermal conductivity are analyzed, which has certain theoretical significance for rapid screening of TE materials with low lattice thermal conductivity. Finally, the efficient ways to obtain the low lattice thermal conductivity for TE property optimization are proposed.