In order to improve the measurement accuracy of dividing error of encoders, the principle and method to measure dividing error by using angular polygon and autocollimator were introduced, and the error sources were analyzed. According to the measurement principle, coordinate systems of angular polygon and autocollimator were established. Utilizing the method of coordinate transformation, precise mathematical models were deduced for indicating the relationships between dividing error and misalignment errors, such as parallelism error between the angular polygon working surface and the axis of the tested unit, perpendicularity error between autocollimator optical axis and the working surface of angular polygon, parallelism error between vertical wire of the autocollimator and the axis of the tested unit. In order to verify the error models of misadjustment, three experiments were performed in the laboratory, taking the positioning error of a single-axis position turntable as test object. The experimental and theoretical results have good consistency and the maximum deviation was less than 0.9″, which indicate that the error models of misadjustment are applicable to guide dividing error measurement.