The shadow of a photovoltaic panel makes the light intensity distribution of the photovoltaic array uneven and reduces the power generation efficiency. It can also produce a hotspot effect, damage photovoltaic cell modules, and cause a system failure. To solve the problems of high target density, large overlap, high cost, and poor real-time performance in the shadow detection of photovoltaic panels, this study proposes a shadow detection algorithm for CRC-RetinaNet photovoltaic panels based on the RetinaNet algorithm. First, cross stage partial structure was used in the feature extraction network to improve the accuracy and detection speed. Second, the feature map was extracted using the recursive feature fusion structure to enhance the feature information of all targets. Third, the activation function of the algorithm was improved to enhance the robustness of the network. Finally, the loss function was changed to CIoU loss to improve the positioning accuracy of the target border regression. The experimental results show that the average detection accuracy of the proposed algorithm is 99.24%, which is improved by 4.02% compared with the original RetinaNet algorithm, and meets the requirements of the real-time detection of photovoltaic panels in the environment.