Addressing regional poverty is the prerequisite for completing the process of building a moderately well-off society in all aspects. As a concentrated area of China's impoverished rural population, the study on the spatial and temporal patterns of poverty in hilly and mountainous areas and its driving mechanism is of great theoretical significance for the transformation and development of poor areas and rural revitalization. Based on the perspective of regional poverty, this study, taking Yudu County of Jiangxi Province as an example, used the standard deviational ellipse, spatial autocorrelation, kernel density estimation, and geographical detector model to describe the geographical pattern of rural poverty and its differentiation characteristics in the county, identify the leading factors of rural poverty, and reveal the driving mechanism of poverty in hilly and mountainous areas. The results demonstrated that the phenomenon of spatial agglomeration exists in the occurrence of rural poverty in hilly and mountainous areas, and the aggregation effect decreases with the proper implementation of poverty alleviation policies. The dominant influencing factors responsible for the spatial differentiation of rural poverty in Yudu County include slope of the terrain, traffic accessibility to town centers from administrative villages, road network density, and the number of public service agencies. As time goes, the influence of natural resource endowment conditions on the spatial differentiation of rural poverty has been weakened while geographic conditions gradually plays a leading role. The leading factors influencing the spatial differentiation of poverty in hilly and mountainous areas varied at different stages. Rural poverty in hilly and mountainous areas has its natural and cultural roots. The formulation and implementation of scientific poverty reduction policies should be based on deepening the theoretical understanding of poverty occurrence mechanism, and promoting the alleviation of regional and individual poverty in a coordinated way, so as to realize the transformation, development, and rural revitalization of poor areas.