In order to ensure the subdivision accuracy of photoelectric rotary encoder in the harsh working conditions, an error compensation method based on HHT(Hilbert-Huang Transform) was proposed. To solve the angle measurement malfunction of encoder system caused by sine vibration, a mathematical model of Moire fringe error signal was proposed according to the malfunction form of encoder mechanical system. EMD(Empirical Mode Decomposition) algorithm was used to get the intrinsic mode function of the error signal. The intrinsic mode function was demodulated and analyzed using Hilbert transform. Moire fringe signal including the malfunction characteristics was extracted. Meanwhile, the fundamental wave time domain frequency of the precise code signal was obtained based on the precise code signal square wave information of the photoelectric rotary encoder. The IMF function envelope components matching with the fundamental wave time domain frequency were extracted. A 24-bit photoelectric rotary encoder was selected as the experimental object. The experimental results show that the dynamic subdivision error peak value of the Moire fringe signal is reduced from 200″ to 1.54″ and the subdivision accuracy improved significantly.