Rotary stage system is a fundamental part of multi-axis machine tools. Therefore, measuring geometric error of rotary stage system is significantly important for geometric error calibration of multi-axis machine tools. A method for measuring geometric error of rotary stage system based on laser tracker is proposed. Homogeneous transformation is applied to establish the mathematical model of geometric error of the rotary stage system which can explicitly describe the relationship between geometric errors and volumetric errors. The coordinates of three noncollinear points on the stage are detected by laser tracker and the volumetric errors of each point can be obtained too. The mathematical model of geometric error is performed inversely to achieve a system of equations which can be solved to obtain the six geometric errors of rotary stage system. This method is used to measure the geometric errors of a rotary stage, and the effectiveness of this method is validated by a comparison experiment.