This study proposes a Harris-SIFT algorithm that combines the color information representation and illumination compensation to resolve the contradiction between matching real time and matching accuracy of the traditional SIFT algorithm. The algorithm calculates the color information compensation before the image is grayed. By using the Gaussian weight function, the illumination compensation amount is obtained, the image contrast is enhanced, and the matching accuracy is improved. The Harris corner detection and SIFT algorithms are combined to improve the system rapidity and eliminate badly characterized feature points, improving the real-time system performance. Further, the experimental results denote that the proposed matching algorithm improves the real-time performance of the algorithm while improving the matching accuracy.