To make the face recognition features learned from the convolutional neural network easier to identify, this paper improves the angular distance loss function A-Softmax by incorporating the facial attributes, such as gender, age, and race, into the training process. By using an attribute-driven loss function and regularizing the feature mapping with attribute proximity, the experimental result shows that more attribute-related discriminating features are learned by the proposed method. The improved algorithm has achieved good results in the face verification datasets, such as LFW, CFP, AgeDB, and MegaFace, verifying the effectiveness of the improved algorithm.