Laser heterodyne interferometer is one kind of nano-metrology systems which has been widely used in industry for high-accuracy displacement measurements. The accuracy of the nano-metrology systems based on the laser heterodyne interferometers can be effectively limited by the periodic nonlinearity. In this paper, we present the nonlinearity modeling of the nano-metrology interferometric system using some adaptive filters. The adaptive algorithms consist of the least mean squares (LMS), normalized least mean squares (NLMS), and recursive least squares (RLS). It is shown that the RLS algorithm can obtain optimal modeling parameters of nonlinearity.