In the process of security inspection, rapid and accurate identification of prohibited items is conducive to maintaining public security. To address the problems of stack deformation, complex background interference, and small-sized contraband detection in X-ray luggage images, an improved model for contraband detection is proposed. This improvement is based on the YOLOX model. First, an attention mechanism was introduced into the backbone network to enhance the ability of the neural network to perceive contrabands. Second, in the neck part, the multi-scale feature fusion method was improved upon, and a bottom-up structure was added after the feature pyramid structure to enhance the performance ability of the network for details, thereby improving the recognition rate of small targets. Finally, the calculation method based on IOU loss was upgraded in view of the disadvantages of the loss function calculation. The weights of various loss functions were also increased according to the characteristics of the contraband detection task, and the punishment of network misjudgment was increased to optimize the model. Upon using the improved model on the SiXray dataset, an mAP of 89.72% was attained and a fast and effective FPS arrival rate of 111.7 frame/s was achieved. Compared with mainstream models, the accuracy and detection speed of the proposed model were improved.