Traffic sign detection is the key in the driving process of intelligent vehicles, which is of great significance to vehicle analysis of road conditions. Aiming at the problems of too many parameters and low accuracy of existing traffic sign detection algorithms, a lightweight traffic sign detection algorithm based on improved real-time detection transformer (RT-DETR) was proposed. Firstly, the ResNet network in the backbone network of the model was replaced by VanillaNet network to reduce the number of layers and parameters. Secondly, the bi-directional feature pyramid network (BiFPN) was used to replace the path aggregation network (PAN) structure in the feature fusion module of RT-DETR to increase the fusion ability of the model and extract more abundant feature information. Finally, the global attention mechanism (GAM) was added to the feature fusion module to enhance the model perception of global information and improve the detection performance of multiple targets and obscured targets. The proposed algorithm was tested on traffic sign datasets. The mean average precision (mAP) value of the improved RT-DETR algorithm reaches 87.7%, 3.6% higher than that of the original algorithm, and the parameters are reduced by 21%, meeting the deployment needs of intelligent vehicle equipment, which proves the effectiveness of the improved algorithm.