The detection of video abnormal behavior is paramount to ensure public safety. In this paper, the abnormal behavior detection algorithm based on deep learning is classified and summarized. First, the overall process of abnormal behavior detection is presented. Then, based on the neural network training method, the development and application of deep learning in the field of abnormal behavior detection are discussed from three aspects: supervised learning, weakly supervised learning, and unsupervised learning, and the advantages and disadvantages of different training methods are analyzed. Finally, commonly used datasets and performance evaluation criteria are presented, the performance of the different algorithms is analyzed, and future directions are discussed.