Accurately extracting the crack characteristics using the traditional crack detection algorithm is challenging owing to the uneven light intensity, complex background, and significant noise interference of concrete pavement. Herein, to improve crack detection accuracy and reduce computational redundancy, a pavement crack detection algorithm was proposed that used an interlaced low-rank group convolution hybrid deep network combined with low-rank kernel and group convolution. First, crack image datasets were established using the overlapping sliding window clipping method. A robust classifier was generated on the training set to classify crack and no-crack images. Then, the adaptive threshold method was used to obtain a crack binary image with clear edge contours. Moreover, the central axis method was used to achieve the maximum width of the crack. The performance of the model was verified on the testing set. Experimental results show that the detection accuracy is 0.9726, thus showing an improvement over the traditional crack detection algorithm. Compared with the convolutional neural network and its variants, the proposed model involved a significantly reduced set of parameters. Images were processed at 14 frames per second, and good detection results were achieved on three public datasets. For crack widths greater than 2.5 mm, the relative error of the calculation is less than 0.02, which complies with practical engineering requirements.