Motion blur restoration is essential for the imaging of moving objects, especially for single-pixel imaging (SPI), which requires multiple measurements. To reconstruct the image of a moving object with multiple motion modes, we propose a novel motion blur restoration method of SPI using geometric moment patterns. We design a novel localization method that uses normalized differential first-order moments and central moment patterns to determine the object’s translational position and rotation angle information. Then, we perform motion compensation by using shifting Hadamard patterns. Our method effectively improves the detection accuracy of multiple motion modes and enhances the quality of the reconstructed image. We perform simulations and experiments, and the results validate the effectiveness of the proposed method.