Due to the ambiguity of anomaly definitions, the scarcity of anomalous data, as well as the complex environmental background and human behavior, video anomaly detection has always been a challenging problem in the field of computer vision. Existing anomaly detection methods based on deep learning often use a trained network to extract features or are trained based on the existing network structure, instead of designing a network for the goal of anomaly detection. In this paper, a new anomaly detection method—Deep Support Vector Data Description (DSVDD) was introduced, which was trained on an anomaly detection based objective. According to DSVDD, not only the smallest size data hypersphere coule be found to establish SVDD, but also useful data feature representations and normal models could be learned. Then, in the testing stage, the samples mapped inside the hypersphere were judged as normal, while the samples mapped outside the hypersphere were judged as abnormal. The method proposed in this paper achieves 87.4% and 74.5% frame-level AUC on the CUHK Avenue and ShanghaiTech Campus datasets, respectively, which outperforms existing state-of-the-art approaches.