To improve the accuracy of small photoelectric encoders without increasing the size, the reason how interpolation errors are generated by the calculation method is analyzed. A new photoelectric encoder fine signals interpolation method based on coordinate rotation digital computer (CORDIC) algorithm is proposed, and it can make use of simple shifting and addition operations to directly interpolate acquired quadrature fine signals and resolve the phases, which can get rid of interpolation errors caused by looking up in the interpolation table. The algorithm is analyzed and improved finally, and the calculation speed is proved while maintaining appropriate accuracy. Moire fringe signals from a 16-bit small photoelectric encoder are interpolated into 256 parts with the suggested algorithm, and the root mean square error is decreased by a half compared to the calculation method. Experimental result shows that the new interpolation method can be directly applied to high-accuracy interpolation of moire fringe signals. It may play an important role in the further research and development of small and high-accuracy photoelectric encoders.