Determining the types of rock is crucial to understand the origin and history of geological units. The recognition and classification of lithologies are usually performed by experienced researchers based on the colors and morphologies with the accuracy not guaranteed, particularly when trying to distinguish between units with similar lithology. Different from the subjective judgment, spectral means can provide multiple optical parameters among different frequencies. Information of the samples can be determined by establishing the relationship between optical parameters and the physical properties, to achieve qualitative and quantitative evaluation. In this study, terahertz (THz) spectroscopy was employed to discriminate the lithology of rock gathered from different regions. Based on the relations between effective refractive index (n) and attenuation coefficient (a), the samples can be classified into four gatherings, with the result in agreement with their lithologies. Besides, the variation tendency of THz parameters was plotted with the componential and structural properties, indicating that the variant THz responses were resulted by varying compositions and structures for different classifications of rocks. The mineral composition is the main factor affecting the absorption and refraction of the THz wave. Element content can be estimated with the THz parameters for Fe and Mg. For sandstone with stable elements, both the absorption and refraction of THz wave are negatively correlated with the porosity. Organic content in oil shale has a reverse effect on THz absorption and refraction, in which the higher oil yield results in a stronger absorption and lower refraction. Herein, the refractive index is positively correlated with the absorption for granite, limestone and sandstone, while the relation between them is negatively correlated for oil shale. Our results prove that the THz technique is a promising means for determining the lithology and petrophysics properties, which will be a significant supplementary in geology research.