Remote sensing images have many problems, such as complex background, small targets, and dense arrangement. The target detection method based on depth learning can improve the accuracy of target detection, but there are many problems, such as more model parameters and general detection speed. Aiming at the above problems, a remote sensing image target detection method based on improved YOLOv4 is proposed. First, the lightweight network Mobile NetV3 is used to replace the original feature extraction network of YOLOv4 to improve the detection speed; second, the self-attention mechanism is concatenated in the prediction layer, and the improved non maximum suppression algorithm is used for post-processing; finally, in the image preprocessing, Mosaic method is used to enhance the data, K-means method is used to obtain the candidate frame parameters that better match the remote sensing target, and Complete Intersection Over Union (CIoU) loss function is used in the prediction layer to locate the coordinate frame. The experimental data set consists of two classical remote sensing datasets, NWPUVHR-10 and DOTA, including 10 categories of ships, vehicles, and ports. The results show that when the input size of remote sensing image is 608×608, the detection speed is 54 frame/s, 1.6 times that of YOLOv4, and the average accuracy is 85.60%. The proposed method reduces the parameter amount and improves the detection speed while maintaining a high detection accuracy.