According to the problem of low recognition accuracy of traditional roughness measurement methods, a roughness detection method based on transfer learning and model fusion was proposed. Firstly, the CCD module in the roughness detection system was used to collect the workpiece surface images and construct a data set. Secondly, through the migration fine-tuning VGGNet-19, Inception-V3 and DenseNet121 multi-model fusion, a suitable roughness detection model is obtained by multi-model fusion. Finally, the data set is used for network training to extract the texture details from the images and achieve accurate recognition of the roughness level. The experimental results show that 15 different roughness level images from turning, milling and grinding are used, and the recognition accuracy of the system can reach up to 91%. The results show that the proposed system can effectively realize the automatic detection of roughness grade.