In order to ensure the measurement accuracy of precise photoelectric encoders in harsh working environment, a kind of automatic compensation system based on the high-resolution digital potentiometer, DSP and CPLD for the Moire fringe photoelectric signal was established. Firstly, the principle and composition of automatic compensation system was introduced. Meanwhile, the compensation algorithms for the components of Moire fringe signal deviation was integrated. And the compensation model for photoelectric signal subdivision error was established. The system hardware design and software design were expounded. The system error of the compensation system was further analyzed. A 24 bit photoelectric encoder was selected as the experimental object to analyze and test the compensation system. The experimental results show that DC level drift, amplitude deviation, orthogonality deviation, and the second, third and fifth harmonic deviation are simultaneously compensated. The actually static subdivision error is reduced to 0.61″. The compensation system can realize the automatic correction of Moire fringe signal subdivision error in the working state.