To compensate of the nonlinearity error of heterodyne interferometers in nanometer measurement, a method for reducing the first harmonic nonlinearity is proposed. Based on the total reflection theory, polarization properties of coated corner-cube retroreflector are analyzed. Then, the effect on the first-harmonic nonlinearity by the polarization properties of reflected light of the cube-corner retroreflector and axial rotation in the motion direction of the measurement cube-corner retroreflector is derived. The simulation results show that the first harmonic nonlinearity is reduced by the axial rotation of the cube-corner retroreflector. When the axial rotation angle is 97°, the first harmonic nonlinearity is the smallest, which is about 0.05 times smaller than the original. When the nonorthogonal error of the two linearly polarized beams is 6°, and the axial orientation rotation angle increases from 0° to 97°, the first harmonic nonlinearity is reduced from 5.30 to 0.30 nm.