Urban geographic space, climate, and transportation system are interrelated, and recently available traffic and spatial big data bring new opportunities for understanding the relationship among them. Urban rail transit is an important transport mode for residents to travel green and relieve traffic congestion in big cities in China. In-depth study of factors that affect the changes in the spatial and temporal distributions of metro passenger flows is conducive to the formulation of reasonable land use and traffic demand management policies, and can also provide a theoretical basis for real-time response to the changes in travel demand under specific weather conditions and optimization of transit service operation. To study the impact of weather conditions on metro usage in densely populated areas, in this research the influence of local weather factors (including temperature, humidity, rainfall and so on) on hourly metro passenger flows was investigated based on metro smart card data and weather data from Nanjing City, China. A time series model—seasonal autoregressive integrated moving average with explanatory variables (SARIMAX)—was developed to investigate the impact of weather conditions on metro passenger flows. It is found that some weather factors such as rainfall have significant influence on metro passenger flows. Except for some special sites (large residential areas and large transportation hubs), the influence of weather factors on metro passenger flows reduces gradually from the city center to suburban areas. The effects of weather conditions on regular metro passengers and irregular metro passengers were explicitly compared in this study. Irregular metro passengers are found more vulnerable to adverse weather conditions than regular metro passengers.