An efficient carrier removal method in off-axis interferometry to retrieve specimen phase distribution was proposed using image rotation and complex encoding. One rotated interferogram was obtained by rotating the specimen interferogram 180°. The specimen interferogram and its rotated interferogram were firstly complex-encoded into a synthetic interferogram. After implementing Fourier transform, the two cross-correlation orders can be separated and then extracted by applying two band pass filters to the spectrum. The results containing phase distribution, carrier information of specimen interferogram and rotated interferogram can be obtained by implementing inverse Fourier transform. Through a division operation, carrier can be removed without complex calculation, phase unwrapping, or prior knowledge of the system. The feasibility and validity of the method were demonstrated by simulations and experiments. The obtained results show that the method can accurately retrieve specimen phase. The method consumes only 23.32% processing time of the original interferogram rotation method when retrieving thin specimen phase.