To obtain three-dimensional full-field measurements of the surface defects of optical elements, a method of digital holographic microscopic scanning imaging is proposed in this study. Based on the numerical reconstruction algorithm of digital holography angle spectrum, the phase distribution of scratches on the surface of optical elements is obtained, and the whole-field measurement of scratches is obtained using the principle of scanning and splicing. Then, a two-dimensional precision scanning component is developed based on the digital holographic micro-experimental device. For the standard scratch having a width of 50 μm and a depth of 50 nm, we measure a width of 49.2 μm and a depth of 48.9 nm. Next, the full-field three-dimensional morphology of the scratch is obtained by splicing. The experimental results show that the defect panorama with larger scale can be obtained by phase stitching. The relative errors of width and depth are 1.6% and 2.2%, respectively.