Ultra high-speed object location has potential prospects in civilian and scientific applications. However, the conventional image sensor has a bottleneck that can not achieve continuous ultra high-speed imaging. The Single-Photon Avalanche Diode (SPAD) image sensor, as a new type of spike image sensor, can realize continuous ultra high-speed imaging to support high-speed moving object location. The background subtraction method is a simple and effective high-speed location method, but the image noise caused by the SPAD image sensor will seriously interfere with the processing effect of the background subtraction method. Therefore, denoise processing is required before locating. This paper proposes an ultra high-speed object location processor for SPAD image sensors. It consists of a Processing Element (PE) array, a Y-feature generator, and a position calculator, and is capable of denoising and object location processing. The SPAD image sensor adopts row rolling exposure to produce N pixel output of one row at once exposure. The output format of the SPAD image sensor is a single-bit spike image containing only “1” and “0”. The processing array contains N processing units, each PE unit contains 9 ALUs, which equals the size of filtering window 3×3, for processing Gaussian filtering and background subtraction methods with the current and earliest frames of spiking image data accessed from its two adjacent PEs. The current frame data, the earliest frame data, and the convolution kernel are calculated by complement logic and XOR gate in each ALU of the PE, and then the ALU results and the data in the memory are accumulated to obtain the filtering result. The object location module outputs a row of X feature vector and a column of Y feature vector at the end of each frame. The position calculator compares the feature vector with their thresholds and outputs the coordinate position of the object. We propose the Gaussian filtering and background subtraction method to remove the multiplication and subtraction operations, which reduces the computational complexity and hardware resource consumption while improving the processing speed significantly. Based on the overlapping sliding window method on impulse images, this paper replaces multiplications with additions merely by refactoring the filtering equation without any hardware overhead. During background subtraction, a static scene without objects is used as the background. The image is accumulated and the Gaussian filtering method is applied to obtain a background image. Then the negative value of the background image is regarded as the initial partial sum accumulation, and Gaussian filtering is applied to the following images containing moving objects based on this initial partial sum. In this way, we implement the background subtraction algorithm without a subtractor in the data path. Fixed pattern noise removal can be performed by simply controlling the relationship between the number of accumulated images of the foreground and background, and reusing the circuit of the background subtraction method to perform fixed pattern noise removal. The object location method determines the coordinates of the four corners by comparing the result of background subtraction with the threshold. The whole process is implemented on the FPGA development board. The experimental results show that the quality of the recovered grayscale image is significantly improved after Gaussian filtering. The processor can process spike images with a resolution of 128×128 at a speed of 100 Kfps, and locate moving objects in it. Compared with other studies focusing on ultra high-speed object location, the processor we proposed reaches a favorable trade-off between the processing algorithm complexity and hardware resource overhead. This paper also compares the performance of the processor with the original na?ve Gaussian filter circuit using the multiplication operation with our optimized processor. The result shows that our optimized processor can be synthesized to a frequency of 31.8% higher than the original one and achieves about 72.4% superiority over the original one on hardware resources. Moreover, our solution also can be applied to other spike image sensors to build a sensing-computing system.