Object detection in remote sensing images is a challenging task in the field of computer vision. Existing remote sensing image detection methods ignore the speed with the aim of improving the accuracy; however, it is also essential to increase the detection speed in real-time detection scenes, such as in resource surveys and maritime rescue. This paper proposes a lightweight target detection network to realize a trade-off between detection accuracy and speed. The design replaces the original backbone network of YOLOv4 with the pruned MobileNetV2. In addition, the ordinary convolution calculation of the feature extraction method is replaced by deep separable convolution to considerably reduce the computational complexity of the model. Finally, the receptive field enhancement module and attention mechanism module are embedded to improve the detection accuracy of the model. The experimental results on the images in the dataset containing the remote sensing images show that the accuracy, as measured by the mean average precision, is 89.80%; further, the model detection speed is 33.4 frame/s. Compared with YOLOv4, the accuracy only decreases by 1.48 percentage points, but the detection speed increases by nearly 1.5 times. Compared with the YOLOv4-Tiny algorithm, the average accuracy is 9.05 percentage points higher. The proposed model successfully meets the trade-off requirements of speed and accuracy. The weight of the model is only 44 MB, which makes it easy to deploy and indicates that it meets the requirements of real-time detection scenarios.