Vector measurement is an important technology for beam testing of antennas and quasi-optical systems in terahertz band. This paper introduces a terahertz vector measurement system based on a high-sensitivity AlGaN/GaN high-electron-mobility transistor (HEMT) terahertz detector integrated with a quasi-optical lens and waveguide together, which reached the noise equivalent power of -113 dBm/Hz in heterodyne mode at 340 GHz. A hardware circuit is established based on the double frequency-down-conversion technique to suppress phase noise in the system. The experimental results indicate that the minimum measurable power is 119 nW and the phase stability is better than 4° of the system. Measurement of the distribution of both terahertz amplitude and phase has been achieved based on this coherent AlGaN/GaN HEMT detector. An arrayed terahertz vector measurement system could be developed based on this work.