The traditional method based on deep learning does not focus on sub-significant information. Therefore, a multi-branch network based on multi-scale attention (MSA) mechanism was proposed to coordinate significant and sub-significant information. Firstly, the proposed algorithm combined the multi-scale feature fusion module (MSFF) with the attention mechanism to get an MSA module. This module enables the network to adaptively adjust the size of the receptive field according to the input information so as to make full use of information of different scales. Additionally, a multi-branch network was established to realize the coordination of global features and multiple local features. Using the MSA module, weighted enhancement of global information and local detail information can be achieved separately, and a more discriminative feature is obtained for final recognition. The experiment results show that the proposed method performs well on multiple datasets.