A target detection algorithm based on improved YOLOv8 is proposed to address the issues of high-missed and false-detection rates, inaccurate target positioning, and inability to accurately identify target categories in remote-sensing image target detection algorithms. To improve the flexibility of the loss function of the model in gradient allocation and adapt to various object shapes and sizes, a boundary box regression loss function is designed, which combines a nonmonotonic focusing mechanism with geometric factors of the boundary box. To expand the receptive field of the model and weaken the influence of the remote-sensing image background on the detection target, a residual global attention mechanism is designed by combining global attention mechanism and residual blocks. To adapt the model to the deformation and irregular arrangement of target objects in remote-sensing images, the C2f module in the YOLOv8 model is improved by incorporating deformable convolution and deformable region-of-interest pooling layers. Experimental results show that on DOTA and RSOD datasets, mean average precision (mAP@0.5) of the improved YOLOv8 algorithm reaches 72.1% and 94.6%, which are better than other mainstream algorithms. It improves the accuracy of remote sensing image target detection and provides a new means for remote sensing image target detection.