There are many problems exist in the process of registration and recognition of keypoints in 3D point cloud, such as finding matching mismatch, large number of mismatched pairs, and the decreasing of registration and recognition precision. A novel removal method of mismatching keypoints is proposed. In the stage of keypoint detection, an edge point detection algorithm is put forward based on the feature that the edge points and their neighbor points are mostly distributed on the same side. The proposed method can remove the keypoints existing in the edge to improve the repeatability and match ability, and reduce the mismatching rate in the feature matching of keypoint. In the stage of feature matching of keypoints, the initial keypoint matching pairs obtained by nearest-neighbor algorithm are matched, a lot of mismatching keypoint pairs can be removed according to the methods of Kmeans and splitting, and the matching rate of keypoints between 3D point clouds can be improved. Experimental results show that a large number of the mismatching keypoint pairs can be removed,which generated by a complete point cloud matching a complete point cloud, a complete point cloud matching a point cloud with clutter and occlusion, and a partial point cloud matching a partial point cloud, lots of mismatching keypoint pairs, and the matching effect of keypoints can be significantly improved. At the same time, the proposed algorithm is more efficient than random sample consensus algorithm in the time consumption, which is a good supplement to the nearest-neighbor algorithm.