The confidence level of rotation angle error test result was reduced by nonparallelism between photoelectric shaft encoder axis and polyhedron one. In order to decrease rotation angle test error, the rotation angle test error caused by nonparallelism between photoelectric shaft encoder axis and polyhedron one was controlled within 1/3-1/5 of photoelectric shaft encoder angle error. The mathematical models of rotation angle test error and Y bias introduced by nonparallelism between photoelectric shaft encoder axis and polyhedron one were established. The simulation results showed that rotation angle test error and Y bias had periodic change with the angle increase, the cycle were π and 2π respectively. When the tilt direction of polyhedron axis was fixed, the bigger angle between encoder axis and polyhedron one, the bigger peak values of angle test error and Y bias, when the angle between photoelectric shaft encoder axis and polyhedron one was fixed, the tilt direction of polyhedron axis only changed the phases of rotation angle test error curve and Y bias one and not changed the shapes of the curves. According to the polyhedron-autocollimator method, the established mathematical models were verified by experiments. Experimental results indicate that the test results and the mathematical models are in good conformity. In the actual test, the rotation angle test error is pretested, the curve is drawn and fitted by the least squares method, the parallelism and tilt direction are obtained, the parallelism is adjusted according to tilt direction until error peak satisfies test requirement.