The mechanisms of frictional chatter of a diamond rulling system and a elastic tool post were researched in mechanical ruling grating processing, and a dynamic model for a mechanical ruling grating was established. Then, the stability of the chatter system was analyzed, and its stability condition and stability threshold were presented. Furthermore, the chatter was researched and analyzed from the energy point of view, and the same stability condition was obtained. A mechanical ruling test were performed with a test device for grating ruling only by changing the ruling speed. After measuring and analyzing the ruling forces, the characteristics of the ruling forces in the vertical direction dropping with the ruling speed were verified, and it satisfied with the precondition for the establishment of frictional chatter. Finally, the surfaces of grooves under four different ruling speeds: 2 mm/s, 6 mm/s, 10 mm/s and 13 mm/s, were measured, which verifies that the chatter will occur when the ruling speed exceeds the critical value (6-10 mm/s). The experiments confirm the effectiveness of the dynamic model of chatter and the existence of the stability threshold. It provides the theoretical foundation for restraining the chatter during grating mechanical ruling process and deep quantitative analysis.