Optical rectification (OR) is one of the most effective methods to generate broadband terahertz (THz) wave. The process of THz wave generation based on OR mainly depends on the interaction between the pump beam and the nonlinear crystal. The OR efficiencies in a 5 mm gallium phosphide (GaP) bulk crystal and a 6 mm GaP waveguide are compared between the central spot of a Bessel beam and a Gaussian beam with the same power. The experimental results show that the OR efficiency of the Bessel beam is 2.04 times larger than that of the Gaussian beam in the GaP bulk crystal, whereas the relative efficiency of the Bessel beam is increased to 3.46 times in the GaP waveguide due to perfect phase matching between the pump beam and the THz wave. The THz wave fields generated by both pumps have a Gaussian distribution, and the THz spectra generated by the Bessel beam have obvious red shifts. The fact that the Bessel beam could improve the OR efficiency will be helpful for achieving high-power and compact THz sources, which is valuable for various THz applications.