In the current process of detecting commercial sand and gravel aggregates, the manual detection is inefficient, greatly affected by subjective factors, and the detection accuracy is not ideal. This paper proposes a convolutional neural network (CNN) based on the sand and gravel aggregate image classification model (CNN13). This classification model refers to the classic CNN visual geometry group 16 (VGG16) model to improve the network structure and optimize parameters. The CNN13 model uses the TensorFlow deep learning framework to build a 13-layer CNN structure. The experimental dataset includes 5000 digital images, which is collected from the sand and gravel aggregates in the daily production of a commercial concrete manufacturer. The model uses GPU for high-speed calculation during the training process. Compared with the VGG16 model, the CNN13 model has fewer convolutional layers and parameters, lower requirements for GPU memory, faster training speed, higher classification accuracy, and classification accuracy for each level of sand and gravel aggregates is more than 99%.