Herein, a facial expression recognition method based on local feature fusion of convolutional neural network (CNN) is proposed to improve recognition rate and real-time performance of facial expression classification. First, a CNN model is constructed to learn the local features of the eyes, eyebrows, and mouth. Then, the local features are sent to a support vector machine multi-classifier to obtain their posterior probabilities. Finally, a particle swarm optimization algorithm is used to optimize the fusion weight of each feature, realize the decision-level fusion with the optimal accuracy rate, and complete the expression classification. Experiments show that the average recognition rates of the method on the CK+ and JAFFE databases are 94.56% and 97.08%, respectively. Compared with other recognition methods, results show that the proposed method has superior performance, improves the recognition rate and robustness, and ensures the real-time performance of the classification.