Facial features extracted by a deep convolutional network are susceptible to background, individual identity, and other factors, which are mixed with unnecessary features that interfere with facial expression recognition. To solve this problem, an attention model-based facial expression recognition algorithm is proposed in this paper. To avoid overfitting, this method is based on a lightweight convolutional neural network. Moreover, the channel attention model and the spatial attention model are employed to strengthen or suppress the feature map elements. A residual learning unit is used to enable the attention model to learn rich features and obtain an excellent gradient flow. In addition, a key area crop scheme for facial expressions is proposed to solve the problem of noise interference in non-expressive regions. The proposed method is validated on two commonly used expression datasets: CK+ and MMI. Experimental results demonstrate the superiority of the proposed method.