In order to quickly and accurately obtain the three-dimensional surface profile of the inner wall of the deep hole structure and analyze the quality of the deep hole processing, a line scanning system based on laser harmonic modulation was proposed, and a reflective optical system that can penetrate deep into the deep hole structure was designed. The point cloud optimization algorithm for harmonic matching through time window filtering was studied. The algorithm used the harmonic modulation phase range to threshold the near-axis scanning area, thereby completing the point cloud data filtering. Experiments were conducted on three different types of deep holes, and the point cloud data was compared with the Handyscan three-dimensional imager. In this paper, the inner wall area of 5 cm×5 cm was quantitatively analyzed, and the three-dimensional point cloud images before and after optimization were compared. The point cloud before optimization obviously contained many stray points, and the comprehensive average deviation was 0.53 mm. After optimization, the noise was effectively suppressed, and the comprehensive average deviation was reduced to 0.12 mm. In the x-axis direction, the average value of the system position deviation was 0.240 mm, and in the y-axis direction, the average value of the system position deviation was 0.228 mm. Since the total amount of point cloud that needs to be calculated was reduced after optimization, the convergence speed had also been improved to a certain extent, and it stabilized above 3000 points, which was about 65.8% of the time before optimization. It can be seen that the system is suitable for the three-dimensional surface inspection of the inner wall of deep holes, and it provided a new idea for deep hole testing and data noise reduction.