Compared with the traditional aerophotogrammetry, the Unmanned Aerial Vehicles (UAV) images have the problems of bigger and more complex geometric distortion and because of the large photographic scale, the texture of images changes little in some features. These factors lead more difficult to matching UAV images, and increase the number of mismatch cognominal points. At first, according to the Harris operator, extract and match feature points. A method for detecting and eliminate the mismatch points was proposed based on the polygon matching. The method considers the reliability of cognominal points matching from distance, angle and weight. Through the experiment, the feasibilitynumber of mismatch points, operation speed and matching accuracy were calculated out in different areas, and the of the method was verified.