A new type of randomly encoded hybrid grating for wavefront testing by quadriwave lateral shearing interferometry is proposed. The randomly encoded hybrid grating which includes an amplitude encoded grating and a phase chessboard is obtained according to the Fraunhofer diffraction principle and the randomly coding method based on the quantization of radiant flux. Compared with the modified Hartmann mask (MHM), only four diffraction orders exist in its diffraction field, and the quadriwave lateral shearing interferometry is better achieved. System parameters such as the incident beam diameter, grating pitch and the distance from the grating to the imaging plane are analyzed and determined. Comparing the far-field intensity distribution and quadriwave lateral shearing interferograms of the randomly encoded hybrid grating with MHM, results show that only four spots with equal intensity exist in the far-field diffraction of the randomly encoded hybrid grating; the shearing interferogram is stable, and it is capable of detecting arbitrary wavefront aberrations.