In the traditional three-dimensional (3D) point cloud registration process, there are some problems such as high registration error, large amount of calculation and time-consuming. Aiming at these problems, a registration and optimization algorithm of key points in 3D point cloud is proposed in this paper. In the key point selection stage, the edge point detection algorithm is proposed to eliminate the edge points, improve the comprehensiveness and repeatability of the feature description of key points, and reduce the registration error of 3D point cloud. In the 3D point cloud registration stage, K-dimensional tree (KD-tree) accelerated nearest neighbor algorithm and iterative nearest point algorithm are used to eliminate key misregistration points in the coarse registration results, reduce the registration errors, and improve the speed and accuracy of 3D point cloud registration. Experimental results show that the algorithm can obtain good registration results under different cloud data. Compared with the traditional 3D point cloud registration algorithm, the average registration rate and the average registration accuracy of the algorithm are improved by 68.725% and 49.65%, respectively.