The Giant Steerable Science Mirror (GSSM) is the tertiary mirror of the future large telescope, Thirty Meter Telescope. The characteristics of jitter were analyzed in the time domain and frequency domain, including the relationship between them. By considering the integral precision, frequency domain integration was chosen as the GSSM dithering data processing method. Simultaneously, the applicability of a high-precision encoder and high precision accelerometer for jitter measurement was compared, and the rationality of using the accelerometer was verified. For the internal vibration of the system, the internal forces were measured by the method of Frequency Response Function. Using the above findings, the jitter detection of the GSSM prototype was carried out. The jitter in the mirror is 0.079 7 μm, around the azimuth axis is 14.143 2 mas, around the tilt axis is 20.747 5 mas, and the internal force was 0.003 3 N.