Graphene-based surface plasmon waveguides (SPWs) show high confinement well beyond the diffraction limit at terahertz frequencies. By combining a graphene SPW and nonlinear material, we propose a novel graphene/AlGaAs SPW structure for terahertz wave difference frequency generation (DFG) under near-infrared pumps. The composite waveguide, which supports single-mode operation at terahertz frequencies and guides two pumps by a high-index-contrast AlGaAs/AlOx structure, can confine terahertz waves tightly and realize good mode field overlap of three waves. The phase-matching condition is satisfied via artificial birefringence in an AlGaAs/AlOx waveguide together with the tunability of graphene, and the phase-matching terahertz wave frequency varies from 4 to 7 THz when the Fermi energy level of graphene changes from 0.848 to 2.456 eV. Based on the coupled-mode theory, we investigate the power-normalized conversion efficiency for the tunable terahertz wave DFG process by using the finite difference method under continuous wave pumps, where the tunable bandwidth can reach 2 THz with considerable conversion efficiency. To exploit the high peak powers of pulses, we also discuss optical pulse evolutions for pulse-pumped terahertz wave DFG processes.