In order that the optical spectrum analyzer can accurately measure narrow spectral signals, the grating is required to distinguish 2 million points in one cycle. A high-resolution grating positioning system based on diffraction grating, brushless direct current motor and photoelectric encoder is designed. A high-precision closed-loop speed control technology based on PID is realized which can drive the rotation of the grating. Meanwhile, the high-precision photoelectric encoder can quickly detect and feedback the angle position of the grating, and the subdivision circuit can further subdivide the output signal of the encoder. The resolution of the output signal is improved from 16000 to 2048000 per cycle, which greatly improves the overall resolution of the grating positioning system. The performance indexes such as the scanning speed, wavelength repeatability and wavelength accuracy of the grating are tested by experiments, and the positioning accuracy and resolution of the grating are verified.