In view of the high complexity of artificial feature extraction in traditional machine learning and the low recognition rate caused by inadequate feature extraction in single convolutional network, a new facial expression recognition method based on ensemble convolutional neural network is proposed. The method is to construct an ensemble network (EnsembleNet) model based on integrating an improved VGGNet-19GP model after VGGNet-19 with a ResNet-18 model. The model first trains a single model on the training set to make the single model reach the optimal experiment. Then the ensemble test is performed on the testing set. The average accuracy of 73.854% and 97.611% are obtained on FER2013 and CK+ datasets, respectively. By comparison with the VGGNet-19GP and ResNet-18 models and other existing methods, it is shown that the ensemble-based facial expression classification method has the advantages of more accurate classification and stronger generalization ability.