Face expressions are affected by factors such as poses, object occlusion, lighting changes, race, gender, and age. Convolutional neural networks are required to learn features more effectively and accurately. AlexNet has low accuracy in expression recognition and strong input image size limitation. In response to these problems, this paper proposes an improved facial expression recognition algorithm for improved AlexNet networks. Introducing multi-scale convolution to the AlexNet network is more suitable for small-scale expression images, extracting feature information of different scales, and cross-connecting feature fusion with higher-level feature information can be realized while the multiple lower-level feature information is transfered downwards, which can reflect the image information more completely and accurately, and construct a more accurate classifier. Because cross-connections will generate parameter expansion, making network training difficult and affecting recognition results. Therefore, we use global average pooling to reduce the dimensionality of low-level feature information, reduce parameters generated by cross-connections, and reduce overfitting. The accuracy of our algorithm on CK+ and JAFFE databases is 94.25% and 93.02%, respectively.