The roundness error separation technique of three-point method was widely used in the separation algorithm for roundness error of part shapes and rotation error of motors. The key of accurate separation in roundness error depended on setting the angles of three displacement sensor probes logically, in order to avoid that the denominator of transfer function became zero. However, during the process of specific implementation, the angle setting error of displacement sensor probes inevitably existed, which made analyzing the influence of angle setting error become much more important. Based on the classical model of three-point method, and then a new model was presented for analyzing the angle setting error of displacement sensor probes. Taking the diameter results in measuring roundness through three-point method as the goal, according to different parameter settings in actual working situations, the error of measuring results caused by the angle setting error of displacement sensor probes was quantitatively analyzed. Lastly, using capacitive displacement sensor to carry out the research on measuring diameter of circular parts, the effectiveness of error analysis was confirmed in this article and the repeatability accuracy of measuring diameter which can be less than 2 滋m.