With the continuous development of the global port and shipping market, inter-port competition is increasingly manifesting as competition at the terminal level. The efficiency of container terminals will significantly affect the efficiency and competitiveness of the whole port supply chain, and then have an important impact on regional economic development. It has grown up to be a challenging task for many countries to measure and understand the efficiency characteristics of container terminals scientifically, especially for developing countries and emerging market countries. Therefore, research on container terminal efficiency has received more and more attention in recent years, but there are the research results based on the geographical perspective. In this paper, 15 main container terminals in the Yangtze River Delta region are chosen as the research object, and their efficiency between 2008 and 2017 is measured by using the super-efficiency SBM model, and then the spatial-temporal evolution characteristics are analyzed. On this basis, the Tobit model is utilized to analyze the key factors affecting the efficiency of container terminals. The result shows that: (1) the overall level of comprehensive technical efficiency of the Yangtze River Delta container terminal is not high, the highest efficiency value is only 0.581, but the overall growth trend remains unchanged. During the study period, the average scale efficiency value of container terminals showed a slowly decreasing trend year by year, while the average pure technical efficiency value maintained a steady growth trend. The gap between them is narrowing year by year. Besides, the principal driving force for the slow growth of comprehensive technical efficiency comes from the improvement of pure technical efficiency. (2) From 2008 to 2017, the change trends of container terminal efficiency in the study region were different, which jointly promote the significant adjustment of the spatial pattern of container terminals efficiency. During the study period, the coefficient of variation of the comprehensive technical efficiency of the delta container terminal fluctuated between 0.159 and 0.318, and the spatial pattern evolution of container terminals efficiency has obvious phase characteristics. Since 2013, this spatial difference has developed towards equalization. At the same time, there are noticeable differences in efficiency evolution between the coastal and the riverside container terminals, and the former performs better. (3) Depending on the regression analysis results of influencing factors, human resources, port collection and distribution conditions, and port functions have a significant positive effect on container terminal efficiency. Also, the relevant indicators of ownership structure and hinterland economy have not passed the model test, and their impact is not significant.