Sudden mass gatherings are detrimental to people's safety. Therefore, it is paramount to conduct effective crowd counting in high-risk areas. Aiming at the problems of multicolumn neural network structure is bloated, redundant information and time consuming, we proposed a crowd counting model based on a single-column deep spatiotemporal convolutional neural network and modified it for video image counting. First, a fully convolutional network (FCN) is added to the feature fusion of dilated convolution and level jump connection to improve the ability of the network to extract features. Then, to reduce the influence of the angle distortion generated by the video surveillance on the counting results, a spatial transformation module is added to the long short-term memory (LSTM) network structure. Further, the residual connection method is used to connect and improve the FCN and associated timing LSTM network to improve the accuracy of the network counting results. Finally, tests are performed on UCSD, Mall, and self-built population data sets. Results show that the crowd counting accuracy and robustness of the model are better compared with other models.