This study aimed to address issues of difficult diagnosis of benign and malignant masses in breast mammogram. For medical imaging, this study proposed a classification method of benign and malignant masses in breast mammogram based on attention mechanism and transfer learning. First, a new network model was built by combining convolutional block attention module (CBAM) and the residual network ResNet50 to improve the ability of the network to extract the features of the mass lesions and enhance specific semantic feature representation. Then, a new transfer learning method was proposed; instead of traditional method using the ImageNet as the transfer learning source domain, the patch data were used as the transfer learning source domain to complete the domain adaptive learning from local mass patch images to global breast mammogram, which can improve the ability of the network to capture pathological features. The experimental results show that the proposed method achieves an area under the receiver operating characteristics curve (AUC) value of 0.8607 in the local breast mass patch dataset and an AUC value of 0.8081 in the global breast mammogram dataset. The results confirm the effectiveness of the proposed classification method.