The factors affecting the resolution of varied line-space grating angular displacement sensor are analyzed. The calculation formula of the sensor resolution is deduced, in addition, the relation between the parameters and the method of improving the resolution are discussed. Besides, it is theoretically analyzed and verified that the diffraction order has influence on the resolution. The relationship between the resolution and the grating space of the varied line-space grating angular displacement sensor is proved, as well as simulated and verified. Theoretical guidance and calculation method are thereby provided for the design and optimization of the varied line-space grating angular displacement sensor.