How to improve the measuring accuracy of a roundness measuring instrument was researched in consideration of the cylindricity error requirement and cylindricity error assessment searching algorithm. To reduce inclination error of a workpiece in measuring cylindricity error, a leveling method using a dual-point vertical layout was proposed and analyzed. According to the direction cosine of the axis of workpiece, the amount of leveling was defined and calculated to overcome the problem brought by manual adjustment technology and the accurate leveling and high precise measurement were completed. As the assessment of cylindricity error is to search for a cylinder axis that statisfies the minimum condition and the Nelder-Mead simplex method has lower convergence precision and convergence rate because it depends on initial solution,so that a combined algorithm with quasi-Newton method and simplex algorithm was proposed to achieve the fast and accurate search for global optimums.The Matlab simulation and experimental testing on classical functions indicate that the combined algorithm improves the convergence rate and convergence precision effectively and the convergence rate has been increased over 50% and convergence precision increases over once as compared with that of Nelder-Mead simplex method, which ensures and improves the measuring precision of workpieces.