Various forms of cracks can easily occur during the construction and use of concrete structures, leading to many security problems. The traditional manual safety detection method not only consumes financial resources and time but also provides no guarantee of accuracy. To improve the efficiency of crack recognition on a concrete surface, a recognition method based on convolutional neural network combined with clustering segmentation is proposed herein, which achieves accurate recognition of concrete surface crack images under more complex backgrounds. Results show that the proposed method can not only efficiently classify but also identify cracks in more complex backgrounds with high accuracy. In addition, the proposed method provides a certain theoretical basis for the workload reduce of crack recognition on concrete surfaces, as well as the maintenance and safety inspection of concrete structures. Furthermore, the proposed method provides references for future fracture-identification studies under higher accuracy and more complex conditions.