Defect detection is important in quality assurance on production lines. This paper presents a fast machine-vision-based surface defect detection method using the weighted least-squares model. We assume that an inspection image can be regarded as a combination of a defect-free template image and a residual image. The defect-free template image is generated from training samples adaptively, and the residual image is the result of the subtraction between each inspection image and corresponding defect-free template image. In the weighted least-squares model, the residual error near the edge is suppressed to reduce the false alarms caused by spatial misalignment. Experiment results on different types of buttons show that the proposed method is robust to illumination vibration and rotation deviation and produces results that are better than those of two other methods.