A high-speed target tracking algorithm with high accuracy and stability is proposed for pulse image sensors. First, the principle of a pulse image sensor is introduced. Second, the traditional visual background extraction (Vibe) algorithm is improved by combining the pulse density characteristics of the sensor to remove the ghost and hole issues in the traditional Vibe algorithm, this further improves the integrity of motion detection. Subsequently, combined with motion detection, the traditional mean shift (MS) tracking algorithm is enhanced to improve the accuracy and stability of target tracking. Finally, scene reconstruction and target tracking are completed via image reconstruction. In the three high-speed scenes experiments, compared with the traditional MS algorithm, which is directly applied to image sequences, the maximum tracking error of the proposed algorithm for high-speed targets reduced from 11.0454 to 2.2361, from 14.1421 to 5.0000, and from 26.1725 to 5.0990, respectively. The position standard deviation of target tracking decreased from 7.9879 to 2.0393, from 12.0790 to 2.7454, and from 14.4591 to 3.5654, respectively. In summary, the proposed algorithm can effectively improve target tracking accuracy and stability and is more suitable for pulse image sensors than the other algorithms.