In order to improve the accuracy of straightness measurement,a sub-pixel edge detection algorithm using an operator named Sobel-Guass fitting was presented in the measurement system based on CCD image using laser beam as straight datum. At first,edge was thinned based on Sobel operator,and then the precision of the image edge location was further improved through Gauss function fitting and interpolation at the gradient direction. At last,the geometrical parameter was attained precisely to subpixel by best circular least-square fitting. Consequently,the precision of measurement system was improved about an order. The experimental results demonstrated that the position accuracy of edge can reach 0.1 pixel after thinned by pixel,and the position accuracy of laser spot center is better than 0.03 pixel.