As for laser polishing of metal materials, the main criterion of laser polishing is surface roughness. In this paper, the orthogonal experiment and the response surface method are used to design an experiment of surface roughness after laser polishing in order to study the effects of defocusing distance, laser power, repetition frequency, and scanning speed on surface roughness. First, four factors and three levels of orthogonal experiments are designed, and the range analysis and comparison are used for searching the best result. Then, the response surface method is used to design a Box-Behnken Design (BBD) experiment with four factors and three levels. The numerical model of surface roughness is established and the optimized laser polishing process parameters are obtained. The minimum surface roughness is 0.1178 μm using the range analysis of orthogonal experiment, slightly higher than the surface roughness of 0.1112 μm using the response surface method. Finally, the best process parameters of laser polishing are listed: defocusing distance is 3 mm, laser power is 29.825 W, repetition frequency is 91.451 kHz, and scanning speed is 1749.794 mm/s. Under the best process parameters, the surface roughness of TC4 alloys after laser micro-polishing is reduced from 0.3247 μm to 0.1112 μm. Good polishing quality and low surface roughness can be obtained by using suitable process parameters.