For the small-size defects of metal workpiece surface and the difficult segmentation of image defects due to non-uniform illumination, an improved U-net semantic segmentation network was proposed to achieve accurate image segmentation of surface defects for metal workpieces. Firstly, the convolutional block attention module (CBAM) was integrated into the U-net netwok to improve the significance of the defective targets in the image. Secondly, part of the traditional convolution in the network was replaced by depthwise over-parameterized convolution (DO-Conv) to increase the number of learnable parameters of the network. Then, the Leaky Relu function was used instead of the partial Relu function in the network to improve the feature extraction ability of the model for the negative intervals. Finally, the median filtering and non-uniform illumination compensation method were used for image preprocessing, so as to reduce the effect of non-uniform illumination on the surface defects of metal workpiece images. The results show that the improved network mean intersection over union, accuracy rate and Dice coefficient index reaches 0.833 5, 0.933 2 and 0.867 4, respectively. The improved network significantly improves the segmentation effect of surface defect images of metal workpieces.