The basic principles of wavefront sensing based on phase diversity (PD) technology are expatiated, and the genetic algorithm (GA) is proposed to be applied in the data post-processing of PD testing aiming at dealing with the multi variables and the highly nonlinearity of merit function. The wavefront error of an off-axis three-mirror anastigmatic (TMA) system is tested with PD technology based on extended target, in which the defocused image is obtained by shifting the image plane. To verify the improvement of image quality and phase estimation of PD technology, a comparison experiment of extended object on high-resolution imaging and phase estimation is carried out based on the PD and interferometric testing. It can be observed that the deviation of root mean square (RMS) between the wavefront error obtained by PD testing and by interferometric testing is 0.0329λ (RMS) for the coarse-aligned off-axis TMA system, and the deviation decreases to 0.013λ (RMS) when the system is well aligned. The precision of PD technology can satisfy the requirement of engineering. The imaging experiments are also carried out and image quality can be well improved according to the wavefront estimated by PD.