The influence of ultrasonic wave excited by a laser point or line source on the detection of rail surface defects is investigated. In addition, the sound fields, acoustic signals and the detection effectiveness of rail surface defects under these two types of excitation modes are compared by the finite element method. The detection of horizontal, vertical, and oblique 45° rail surface defects is simulated by the finite element method and the corresponding experimental test is also conducted. The consistency between experimental and simulated results shows that the acoustic surface wave excited by a laser line source possesses good directionality and a high signal amplitude, which is more conducive to the detection of rail surface defects and has a good detection ability for different types of rail surface defects.