As a research hotspot in the field of computer vision, expression recognition plays an important role in emotion recognition, human-computer interaction, intelligent security and other fields. Aiming at the problem that VGG19 is easy to overfit due to the large number of fully connected layer parameters in the process of data training, CapsNet is used to replace the fully connected layer of VGG19 to realize the cascade of VGG19 and CapsNet, so as to improve the problem of overfitting during training. At the same time, the accuracy of the cascaded model on the RAF-DB dataset is improved by 5.28%. In view of the problem that the MaxPool of VGG19 feature extraction network is easy to lose information of face feature map, the SoftPool is used to replace MaxPool, so as to retain the fine-grain features of face to the maximum extent. The experiments show that the accuracy of the two improved models is 84.21% on the RAF-DB dataset and 73.16% on the FER2013 dataset, which has a good expression recognition effect.