Current methods of expression recognition based on deep learning exhibit problems such as a large number of parameters and poor real-time performance. To resolve these problems, this paper proposes a facial expression recognition method by merging the multilayer features of a lightweight convolutional network. First, the improved inverted residual network was considered as the basic unit for building the lightweight convolutional network model. Then, the shallow features associated with the convolutional network subjected to the methods of pooling, 1×1 convolution, and global average pooling were merged with the deep features of the network for expression recognition tasks. The proposed method was verified by employing RAF-DB and AffectNet, which are commonly used expression datasets. Results indicate that the facial expression recognition accuracies of the proposed method when considering the RAF-DB and AffectNet datasets are 85.49% and 57.70%, respectively. Furthermore, the number of model parameters is only 0.2 × 10 6.