The crack is a common disease on bridges and roads. Aiming at the problem that its detection accuracy needs to be improved, a bridge cracks detection algorithm based on Mask region-based convolutional neural networks (RCNN) was proposed, and a semantic enhancement module (SEM) was designed. Combined this module with feature pyramid network (FPN), a new multi-scale feature map was obtained by feature fusion. In view of the complexity and diversity of crack forms and the difficulty of identification, the cracks were divided into two categories for detection, and two strategies were formulated for comparative experiments. The results show that the improved method can get better detection results, the detection accuracy can reach to 99.8%, and the mean average precision (mAP) can be improved by 12.6%.