For the complex background of remote sensing images, the key objects in a scene image are small and large-scale variations, so that it needs to improve model representation ability for scene classification. Therefore, a deep multi-branch feature fusion network is proposed for remote sensing image scene classification. The multi-branch network structure is utilized to extract high-level, middle-level and low-level feature, and the three levels of features are then split-fused-aggregated into a grouped fusion. The fusion method is based on the proposed split-fusion-aggregation group fusion method. Finally, in order to pay attention to the loss of difficult to distinguish samples and labels, a loss function is proposed. The experimental results proved that the method proposed in this paper is very effective for improving the accuracy of classification. The accuracy rate on the UCM, AID, and OPTIMAL datasets surpasses other state-of-art algorithms. On the UCM dataset, 50% of the samples are trained, the accuracy rate is 99.29%, and the classification accuracy rate is increased by 1.35% compared with ARCNet-VGG16 algorithm. On the dataset AID, 50% of the samples are trained, and the accuracy rate is 95.56%, an increase of 0.98% compared with Two-Stream algorithm. 80% of the samples are trained on the dataset OPTIMAL, and the accuracy rate reached 95.43%, with an improvement of 2.73% compared with ARCNet-VGG16 algorithm.