An improved YOLOv5 network model is proposed to resolve high overlap, heavy occlusion, and complex background interference in X-ray luggage image security detection by introducing the convolutional block attention module, data enhancement strategy, and the weighted boxes fusion algorithm for X-ray prohibited item detection. The convolutional block attention module is introduced in the Neck to enhance the extraction of deep important features and suppress background interference of X-ray prohibited items features. The Mixup data augmentation strategy is employed during the training process to simulate the detection scene with high overlap and heavy occlusion items to strengthen the learning ability of the model for complex samples. During the testing process, the weighted boxes fusion algorithm is used to optimize the redundant prediction boxes to enhance its prediction accuracy. The proposed model is tested on three large-size complex datasets (SIXray, HiXray, and OPIXray), resulting in mean average precision values of 89.6%, 83.1%, and 91.6%, respectively. The results show that the proposed model can effectively improve the ability of YOLOv5 in detecting complex contrabands. The proposed model performs better than many current advanced algorithms, indicating its high accuracy and robustness.