In order to improve the subdivision accuracy of photoelectric rotary encoder, the method of automatic compensation of sine deviation for the grating fringe photoelectric signal is proposed. According to square wave information from precise code conversion, adaptive sampling of precise code signal is realized. The transformation process of precise code time domain signal to equal interval spatial domain signal is completed. Periodic grating fringe photoelectric signal of 24 bit photoelectric encoder is actually collected and frequency spectral analysis of discrete amplitude sequence for photoelectric signal is completed. The main harmonic components of sine deviation for high precision photoelectric encoder signal are revealed and waveform equation is established. Based on the mathematical model of grating fringe photoelectric signal and amplitude subdivision principle, compensation model of subdivision error is established. Seven undefined parameters of signal model are identified using particle swam optimization (PSO) and the signal is corrected. Uncompensated and compensated subdivision error are analyzed using subdivision error compensation model. The experimental results show that the peak value of subdivision error is reduced from 0.923″ to 0.316″. This method can actually be used in encoder system, which improves environmental adaptability and angle measurement reliability of the encoder.