Breast cancer computer-aided diagnosis (CAD) system is playing more and more important role in medical detection and diagnosis. In order to classify tumor and non-tumor in magnetic resonance imaging (MRI), a novel breast cancer CAD system based on deep learning and transfer learning is designed. First, we balance the imbalanced data sets and use data augmentation to deal with it. Then, we use the convolutional neural network (CNN) to extract CNN features from MRI data sets, use the same support vector machine to evaluate the feature extraction abilities of different layers, and select the highest F1 score layer as the node of fine-tuning, the layers behind it, which has relatively low dimension as the node of connection of new networks. Next, we select the newly designed fully-connected layers with two layers to form a new network, and use transfer learning to load weights on the new network. At last, we freeze the layers before the node of fine-tuning, while other layers can be trained in the fine-tuning procedure. The CAD systems are built on three CNN networks, including VGG16, Inception V3, and ResNet50. The effects of the system based on VGG16 and ResNet50 have the best performance, and twice transfer learning can improve the performance of VGG16 network system.