In order to solve the problem that the accuracy of defect detection algorithm is not high and it is difficult to meet the actual requirements due to the complex surface texture of aluminum profile and the large difference in defect size, an improved object detection network AM-YOLOv3 (attention-guided multi-scale fusion YOLOv3) is proposed. The attention guide module and four prediction scales are designed to realize the multi-scale feature extraction of aluminum profile surface defects. A bottom-up feature transmission path is constructed, which is combined with the original feature pyramid network to form a twin-towers structure, and the multi-scale feature fusion is realized. K-medians algorithm is used for anchor box clustering, which more accurately characterizes the distribution law of anchor frame size and improves the convergence speed of the network. The performance of the proposed algorithm is verified by experiments on the public aluminum profile dataset. The experimental results show that the mAP (mean average precision) of the proposed algorithm reaches 99.05%, which is 6.8% higher than that of the YOLOv3 model, and the frame rate reaches 43.94 frame/s.