In recent years, the rapid development of high-resolution remote sensing technology has provided an effective technical means for detecting ground objects along the railway. The regression-based one-stage target detection method YOLOv4 has the advantages of high detection accuracy and fast speed. However, when it is used for remote sensing image detection, small targets are still missed due to the loss of some detailed feature information, and large-area ground object detection. Due to the problem of low efficiency, this paper improves the YOLOv4 network model to detect ground objects along the railway in remote sensing images. This paper improves the YOLOv4 network model to detect the ground features in remote sensing images along the railway. First, the CBM (Convolution Batch Normalization Mish) module is designed with composing of convolution, batch normalization, and Mish activation, and the DCBM (Double CBM) module is used for the transmission layer of the densely connected network (DenseNet) for the YOLOv4 network feature extraction. It can achieve feature transfer and information reuse and enhance small target feature detection capabilities. Then, to address the defects of YOLOv4 in the inefficiency of large area detection and the large space of model parameters, the SE (Squeeze Excitation) channel attention mechanism is used after each residual cell of Cross Stage Partial (CSP) in the backbone network to reduce the number of repeated calls of the SE attention module. Hence the performance of the network is improved while reducing the number of model parameters and improving detection efficiency. Finally, for the problem of difficult extraction of railroad targets in images, an improved channel space attention mechanism ICBAM (Improved Convolutional Block Attention Module) is introduced before the network result output to retain the original feature information. It can solve the problem of poor feature extraction ability of railroad targets, and improve the detection efficiency of large-scale targets. To verify the effectiveness of the proposed method, 1 676 remote sensing image samples are selected along a particular section of the railway with a resolution of 1 920×1 080. Railways, houses, buildings, farmland, and ponds in the data set are selected as targets for inspection, and some current popular target detection methods are compared. The experimental results show that the improved method enhances the detection ability of small targets, improves the accuracy and speed of detection, and improves the detection efficiency of large-scale targets. Compared with the YOLOv4 algorithm, the improved method mAP has increased by 2.11%, accuracy increased by 2.93%, the recall rate has increased by 3.79%, and the model size is reduced by 8.53%. The proposed method also provides an effective method for rapidly and accurately detecting ground objects in remote sensing images along the high-speed railway.