Two methods for determining the residual nonlinear error of a dual-frequency interferometer are proposed. The first is an estimation method, which can get the nonlinearity error based on the amplitude modulation of the measuring signal when moving the reference mirror of the interferometer continuously. This method has the advantages of easy operating and high resistance to the environment influence. The second is nanometer calibration, which can trace the measurement result to meter definition. In this paper, a dual-frequency interferometer named SJD5 with nanometric capability is introduced.Nanometer calibration between SJD5 and a F-P interferometer of the National Institute of Metrology (NIM ) is also described in detail. The residual nonlinearity of SJD5 is determined as 1. 4 nm by the estimated method and less than 2. 5 nm by nanometer calibration. These two results fit well with each other under consideration of environment influence.