Based on the displacement measurement system of grating interference fringes, this paper proposed a fringe image displacement detection algorithm based on gray interpolation, i.e. grayscale extreme value algorithm. First, the optical displacement measurement system is introduced. Then, the principle of grayscale extreme value algorithm is explained. In order to verify this algorithm, the fringe image of the optical system was simulated and the algorithm simulation was used to locate the image at sub-pixel level. The traditional image correlation method and the optical flow method were compared and analyzed, and the relative error curve was obtained. The proposed algorithm accuracy is 1.797 pixel. Finally, a grating interferometric optical system was built, and the moving stripe data were collected and demodulated using the grayscale extreme value algorithm to realize sub-pixel level displacement demodulation with an experimental accuracy of 1.093 pixel under a large range. The experimental results show that the grayscale extreme value algorithm possesses a high measurement accuracy in a large range and can be used to realize sub-pixel level positioning. Moreover, it possesses the characteristics of short calculation time, little influence of noise, and strong robustness. The proposed method is also suitable for image-based displacement signal demodulation of other systems, such as self-coherent moire fringe system and photoelectric shaft encoder.