A novel method with high resolution has been proposed which is based on heterodyne interferometry and scanning probe to get the measurement values of superfinish surface scratch. But the optical nonlinear errors affect the measurement precision of the system. We adopt the Jones matrix to analyze the change of elliptic polarization caused by the composite error factors, which are polarization ellipticity and installation orientation error of laser source, phase retardation of quarter-wave plate and polarization leakage of polarization beam splitter. The measurement errors of frequency mixing about these elliptic polarization beams arriving at the photodiode detector are studied by vector theory. The results show that the measurement errors are periodic errors, and they are more affected by nonideal laser and polarization beam splitter. Also, the methods of reducing nonlinear error according to the numerical results in the system are suggested.