High-precision edge detection of fiber and waveguide chip is an important basis for the position and posture adjustment of waveguide alignment. Considering that the conventional edge line detection algorithms can not meet the need of high-precision detection of waveguide devices edge, a new fitting method of linear approximation based on the idea of least squares was introduced, where the coefficient converged to zero. First of all, factor analysis of relationship between the fitting function coefficient and the linear correlation coefficient was undertaken, and a assessment model of non-random error was established. Then, under the hypothesis that non-random errors was assumed to be large enough and was limited in small enough regions, a subspace, which belonged to a general sample space and only have random errors, was proved to have less variance than that of the general sample space. Finally, based on the relationship of residuals of the linear approximation, a new fitting method was proposed. The experiment results showed that the new algorithm accurately detected the linear edge which was damaged and the angle error was less than 0.01 degree. Therefore, the new method has some excellent characteristics, e.g. high precision, anti-impact ability on errors, and fast convergence of fitting coefficients, and it can meet the need of high-precision fields, such as auto micro-electronics and opto-electronics packaging.