Due to the traditional machine vision algorithms there are small feature defect targets on the surface of integrated circuit chips are not obvious to detect, the detection speed is slow and the false detection rate is more. Based on the above problems from both accuracy and efficiency considerations, an improved Faster RCNN namely VanillaNet-11 minimalist network model is proposed as the backbone architecture, avoiding the traditional algorithm ResNet-50 to bring deep and complex link and attention mechanism problems, while further improving the accuracy. The original ROI Pooling layer is replaced by ROI Align to quantify the mismatch problem. Finally, the original NMS is improved by PSRR (Pyramid Shifted with Relationship Recovery) -Maxpool NMS method, which achieves the effect of suppressing the redundant overlapping candidate frames faster while guaranteeing the accuracy. The experimental results show that the average accuracy of the improved network reaches more than 95% in the collected chip defect dataset, which is about 25% higher than that of the original Faster RCNN network, and effectively improves the detection capability of small defective targets.