Aiming at the transmission and storage requirements of three-dimensional model in the large data environment, a three-dimensional point cloud lossy compression algorithm based on the octree is presented. The stop condition of the octree segmentation is improved, so the segmentation can be stopped at an appropriate depth, and the proper size of voxel is ensured. At the same time,the K neighborhood is established based on the segmentation and the outliers of original point cloud are removed by simple and effective statistical method. In the data structure, each node is assigned to a bit mask. The data query and manipulation are traversed by manipulating the bit mask. Then the subsequent point position coding are optimized. The proposed algorithm effectively removes the outliers and miscellaneous points on the surface, and improves the efficiency of point cloud compression in range encoding. The experimental results show that this algorithm can preserve the key information of three-dimensional point cloud data more completely, obtain a good compression rate and shorten compression time.