Real-time and accurate localization and identification of target such as aircraft, ships and vehicles in aerial images is the fundamental basis for further decision-making. Aiming at the issues of low efficiency and accuracy in detecting small targets in aerial images, this paper proposes an improved RT-DETR based small target detection algorithm in aerial images. Firstly, an efficient CCFM-P2ASF scale sequence feature fusion module is constructed to obtain more semantic information and improve the sensitivity to small targets. Secondly, it integrates more flexible learnable positional coding to provide clearer position definition, and then, a more efficient bounding loss function is designed to reduce the deviation of target location prediction and provide more accurate bounding information. Finally, an EMA reparameterization response module is designed to extract input image features more efficiently. The experimental results show that the model size of the improved RT-DETR is reduced by 38.3%, the accuracy, mAP50 and mAP50∶95 is increased by 5.1, 5.0, and 2.2 percentage points respectively compared with the original model. The proposed algorithm achieves better detection performance compared with other mainstream algorithm models.