A series of molecular dynamics (MD) simulation are performed to study the surface deformation behavior of silicon substrate scratched by an atomic force microscopy (AFM) probe. A modified MD model is established, a quantitative index is proposed to describe the pile distribution, and the structure recognition algorithm is used to reveal the generation process of non-crystal layer. On these bases, the effects of scratching velocity, tip radius and probe wedge angle on the scratching process are investigated. Results show that (1) The scratching velocity has little effect on the groove surface. The piles on substrate surface are the least when the scratching velocity is less than 0. 3 nm/ps or greater than or equal to 1.5 nm/ps. (2) The probe wears and tears when the tip radius is less than or equal to 1 nm. The probe deforms elastically when the tip radius is greater than or equal to 1.5 nm. The tip radius should be 2-3.5 nm for the best scratching results. (3) The large wedge angle helps to reduce the piles distributed on the substrate surface.