Review of Semantic Segmentation of Point Cloud Based on Deep Learning

Jiaying Zhang; Xiaoli Zhao; Zheng Chen

doi:10.3788/LOP57.040002

[1] Chang A X, Funkhouser T, Guibas L et al. -12-09)[2019-06-30]. https:∥arxiv.gg363., site/abs/1512, 03012(2015).

[2] Yu H S, Yang Z G, Tan L et al. Methods and datasets on semantic segmentation: a review[J]. Neurocomputing, 304, 82-103(2018).

[3] Zhu Y K, Mottaghi R, Kolve E et al. Target-driven visual navigation in indoor scenes using deep reinforcement learning. [C]∥2017 IEEE International Conference on Robotics and Automation (ICRA), May 29-June 3, 2017, Singapore. New York: IEEE, 3357-3364(2017).

[4] Qi C R, Liu W, Wu C X et al. Frustum PointNets for 3D object detection from RGB-D data. [C]∥2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 18-23, 2018, Salt Lake City, UT, USA. New York: IEEE, 918-927(2018).

[5] Nguyen A, Le B. 3D point cloud segmentation: a survey. [C]∥2013 6th IEEE Conference on Robotics, Automation and Mechatronics (RAM), November 12-15, 2013, Manila, Philippines. New York: IEEE, 225-230(2013).

[6] Garcia-Garcia A, Orts-Escolano S, Oprea S et al. A survey on deep learning techniques for image and video semantic segmentation[J]. Applied Soft Computing, 70, 41-65(2018).

[7] Zhang X F, Liu J, Shi Z S et al. Review of deep learning-based semantic segmentation[J]. Laser & Optoelectronics Progress, 56, 150003(2019).

[8] Li Y, Tong G F, Yang J C et al. 3D point cloud scene data acquisition and its key technologies for scene understanding[J]. Laser & Optoelectronics Progress, 56, 040002(2019).

[9] Tian Q H, Bai R L, Li D. Point cloud segmentation of scattered workpieces based on improved Euclidean clustering[J]. Laser & Optoelectronics Progress, 54, 121503(2017).

[10] Zhan Q M, Yu L, Liang Y B. A point cloud segmentation method based on vector estimation and color clustering. [C]∥The 2nd International Conference on Information Science and Engineering, December 4-6, 2010, Hangzhou, China. New York: IEEE(2010).

[11] Schnabel R, Wahl R, Klein R. Efficient RANSAC for point-cloud shape detection[J]. Computer Graphics Forum, 26, 214-226(2007).

[12] Yang B S, Dong Z. A shape-based segmentation method for mobile laser scanning point clouds[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 81, 19-30(2013).

[13] Li R Z, Liu Y Y, Yang M et al. Three-dimensional point cloud segmentation algorithm based on improved region growing[J]. Laser & Optoelectronics Progress, 55, 051502(2018).

[14] Vo A V, Truong-Hong L, Laefer D F et al. Octree-based region growing for point cloud segmentation[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 104, 88-100(2015).

[15] Huang L L, Li W G, Yang Q L et al. Segmentation algorithm of three-dimensional point cloud data based on region growing[J]. Applied Mechanics and Materials, 741, 382-385(2015).

[16] Charles R Q, Hao S, Mo K C et al. PointNet: deep learning on point sets for 3D classification and segmentation. [C]∥2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 21-26, 2017, Honolulu, HI, USA. New York: IEEE, 77-85(2017).

[17] Qi C R, Yi L, Su H et al. PointNet++: deep hierarchical feature learning on point sets in a metric space. [C]∥Advances in Neural Information Processing Systems, December 4-9, 2017, Long Beach, CA, USA. Canada: NIPS, 5099-5108(2017).

[18] Li Y Y, Bu R, Sun M C et al. PointCNN: convolution on X-transformed points. [C]∥Advances in Neural Information Processing Systems, December 3-8, 2018, Montréal, Canada. [S.l.]: NIPS, 820-830(2018).

[19] Paoletti M E, Haut J M, Plaza J et al. A new deep convolutional neural network for fast hyperspectral image classification[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 145, 120-147(2018).

[20] Wang Q, Gao J Y, Yuan Y. A joint convolutional neural networks and context transfer for street scenes labeling[J]. IEEE Transactions on Intelligent Transportation Systems, 19, 1457-1470(2018).

[21] Mou L C, Ghamisi P, Zhu X X. Deep recurrent neural networks for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 55, 3639-3655(2017).

[22] Kong W C, Dong Z Y, Jia Y W et al. Short-term residential load forecasting based on LSTM recurrent neural network[J]. IEEE Transactions on Smart Grid, 10, 841-851(2019).

[23] Zeng Z Q, Li Z H, Cheng D et al. Two-stream multirate recurrent neural network for video-based pedestrian reidentification[J]. IEEE Transactions on Industrial Informatics, 14, 3179-3186(2018).

[24] Zhong P, Gong Z Q, Li S T et al. Learning to diversify deep belief networks for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 55, 3516-3530(2017).

[25] Li Z Q, Cai X, Liu Y et al. A novel Gaussian-Bernoulli based convolutional deep belief networks for image feature extraction[J]. Neural Processing Letters, 49, 305-319(2019).

[26] Mo K, Zhu S, Chang A X et al. PartNet: a large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding. [C]∥IEEE Conference on Computer Vision and Pattern Recognition, June 16-20, 2019, Long Beach, CA, USA. New York: IEEE, 909-918(2019).

[27] Mian A S, Bennamoun M, Owens R. Three-dimensional model-based object recognition and segmentation in cluttered scenes[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28, 1584-1601(2006).

[28] Yi L, Shao L, Savva M, segmentation from ShapeNet Core55[J/OL] et al. -10-27)[2019-06-30]. https:∥arxiv.gg363., site/abs/1710, 06104(2017).

[29] Armeni I, Sener O, Zamir A R et al. 3D semantic parsing of large-scale indoor spaces. [C]∥2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 27-30, 2016, Las Vegas, NV, USA. New York: IEEE, 1534-1543(2016).

[30] Dai A, Chang A X, Savva M et al. ScanNet: richly-annotated 3D reconstructions of indoor scenes. [C]∥2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 21-26, 2017, Honolulu, HI, USA. New York: IEEE, 2432-2443(2017).

[31] Hackel T, Savinov N, Ladicky L, Remote Sensing, Spatial Information Sciences et al. IV-, 1/W1, 91-98(2017).

[32] Gaidon A, Wang Q, Cabon Y et al. Virtual worlds as proxy for multi-object tracking analysis. [C]∥2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 27-30, 2016, Las Vegas, NV, USA. New York: IEEE, 4340-4349(2016).

[33] Geiger A, Lenz P, Stiller C et al. Vision meets robotics: the KITTI dataset[J]. The International Journal of Robotics Research, 32, 1231-1237(2013).

[34] Zhang R, Candra S A, Vetter K et al. Sensor fusion for semantic segmentation of urban scenes. [C]∥2015 IEEE International Conference on Robotics and Automation (ICRA), May 26-30, 2015, Seattle, WA, USA. New York: IEEE, 1850-1857(2015).

[35] Ros G, Ramos S, Granados M et al. Vision-based offline-online perception paradigm for autonomous driving. [C]∥2015 IEEE Winter Conference on Applications of Computer Vision, January 5-9, 2015, Waikoloa, HI, USA. New York: IEEE, 231-238(2015).

[36] Hu W, Huang Y Y, Wei L et al[J]. Deep convolutional neural network for hyperspectral image classification Journal of Sensors, 2015, 1-12.

[37] Su H, Maji S, Kalogerakis E et al. Multi-view convolutional neural networks for 3D shape recognition. [C]∥2015 IEEE International Conference on Computer Vision (ICCV), December 7-13, 2015, Santiago, Chile. New York: IEEE, 945-953(2015).

[38] Boulch A, Guerrv J, Le Saux B et al. SnapNet: 3D point cloud semantic labeling with 2D deep segmentation networks[J]. Computer & Graphics, 71, 189-198(2018).

[39] Guerry J, Boulch A, Saux B L et al. SnapNet-R: consistent 3D multi-view semantic labeling for robotics. [C]∥2017 IEEE International Conference on Computer Vision Workshops (ICCVW), October 22-29, 2017, Venice, Italy. New York: IEEE, 669-678(2017).

[40] Lateef F, Ruichek Y. Survey on semantic segmentation using deep learning techniques[J]. Neurocomputing, 338, 321-348(2019).

[41] Maturana D, Scherer S. VoxNet: a 3D Convolutional Neural Network for real-time object recognition. [C]∥2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), September 28-October 2, 2015, Hamburg, Germany. New York: IEEE, 922-928(2015).

[42] Wu Z R, Song S R, Khosla A et al. 3D ShapeNets: a deep representation for volumetric shapes. [C]∥2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 7-12, 2015, Boston, MA, USA. New York: IEEE, 1912-1920(2015).

[44] Verdoja F, Thomas D, Sugimoto A. Fast 3D point cloud segmentation using supervoxels with geometry and color for 3D scene understanding. [C]∥2017 IEEE International Conference on Multimedia and Expo (ICME), July 10-14, 2017, Hong Kong, China. New York: IEEE, 1285-1290(2017).

[45] Li Y Y, Pirk S, Su H et al. FPNN: field probing neural networks for 3D data. [C]∥Advances in Neural Information Processing Systems, December 5-10, 2016, Barcelona, Spain. Canada: NIPS, 307-315(2016).

[46] Riegler G, Ulusoy A O, Geiger A. OctNet: learning deep 3D representations at high resolutions. [C]∥2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 21-26, 2017, Honolulu, HI, USA. New York: IEEE, 6620-6629(2017).

[47] Klokov R, Lempitsky V. Escape from cells: deep Kd-networks for the recognition of 3D point cloud models. [C]∥2017 IEEE International Conference on Computer Vision (ICCV), October 22-29, 2017, Venice, Italy. New York: IEEE, 863-872(2017).

[48] Tatarchenko M, Dosovitskiy A, Brox T. Octree generating networks: efficient convolutional architectures for high-resolution 3D outputs. [C]∥2017 IEEE International Conference on Computer Vision (ICCV), October 22-29, 2017, Venice, Italy. New York: IEEE, 2107-2115(2017).

[49] Tchapmi L, Choy C, Armeni I et al. SEGCloud: semantic segmentation of 3D point clouds. [C]∥2017 International Conference on 3D Vision (3DV), October 10-12, 2017, Qingdao, China. New York: IEEE, 537-547(2017).

[50] Le T, Duan Y. PointGrid: a deep network for 3D shape understanding. [C]∥2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 18-23, 2018, Salt Lake City, UT, USA. New York: IEEE, 9204-9214(2018).

[51] Hua B S, Tran M K, Yeung S K. Pointwise convolutional neural networks. [C]∥2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 18-23, 2018, Salt Lake City, UT, USA. New York: IEEE, 984-993(2018).

[52] Jaderberg M, Simonyan K, Zisserman A et al. Spatial transformer networks. [C]∥Advances in Neural Information Processing Systems, December 7-12, 2015, Montreal, Quebec, Canada. [S.l.]: NIPS, 2017-2025(2015).

[53] Huang Q G, Wang W Y, Neumann U. Recurrent slice networks for 3D segmentation of point clouds. [C]∥2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 18-23, 2018, Salt Lake City, UT, USA. New York: IEEE, 2626-2635(2018).

[54] Li J X, Chen B M, Lee G H. SO-Net: self-organizing network for point cloud analysis. [C]∥2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 18-23, 2018, Salt Lake City, UT, USA. New York: IEEE, 9397-9406(2018).

[55] Liu A B, Yang Y Q, Sun Q Y et al. A deep fully convolution neural network for semantic segmentation based on adaptive feature fusion. [C]∥2018 5th International Conference on Information Science and Control Engineering (ICISCE), July 20-22, 2018, Zhengzhou, China. New York: IEEE, 16-20(2018).

[56] Zhong H J, Hu B F. Image semantic segmentation based on high level features fusion[J]. Machine Building & Automation, 48, 178-181(2019).

[57] Hu T, Li W H, Qin X X. Semantic segmentation of polarimetric synthetic aperture radar images based on multi-layer deep feature fusion[J]. Chinese Journal of Lasers, 46, 0210001(2019).

[58] Jiang M Y, Wu Y R, Zhao T Q et al. -11-24)[2019-06-30]. https:∥arxiv.gg363., site/abs/1807, 00652(2018).

[59] Xu Y F, Fan T Q, Xu M Y et al. SpiderCNN: deep learning on point sets with parameterized convolutional filters. [C]∥IEEE Conference on Computer Vision and Pattern Recognition, September 8-14, 2018, Munich, Germany. New York: IEEE, 87-102(2018).

[60] Ma Y X, Guo Y L, Lei Y J et al. 3DMAX-net: a multi-scale spatial contextual network for 3D point cloud semantic segmentation. [C]∥2018 24th International Conference on Pattern Recognition (ICPR), August 20-24, 2018, Beijing, China. New York: IEEE, 1560-1566(2018).

[61] Komarichev A, Zhong Z, Hua J. A-CNN: annularly convolutional neural networks on point clouds. [C]∥IEEE Conference on Computer Vision and Pattern Recognition, June 16-20, 2019, Long Beach, CA, USA. New York: IEEE, 7421-7430(2019).

[62] Ye X Q, Li J M, Huang H X et al. 3D recurrent neural networks with context fusion for point cloud semantic segmentation. [C]∥European Conference on Computer Vision, September 8-14, 2018, Munich, Germany. New York: IEEE, 403-417(2018).

[63] Wang X J, Ma J, Wang N N et al. Deep learning model for point clouds classification based on graph convolutional network[J]. Laser & Optoelectronics Progress, 56, 211004(2019).

[64] Wang Y, Sun Y B, Liu Z W et al. -06-11)[2019-06-30]. https:∥arxiv.gg363., site/abs/1801, 07829(2019).

[65] Zhang K G, Hao M, Wang J et al. -08-06)[2019-06-30]. https:∥arxiv.gg363., site/abs/1904, 10014(2019).

[66] Huang G. Liu Z, van der Maaten L, et al. Densely connected convolutional networks. [C]∥2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 21-26, 2017, Honolulu, HI, USA. New York: IEEE, 2261-2269(2017).

[67] Te G, Hu W, Guo Z M et al. -06-08)[2019-06-30]. https:∥arxiv.gg363., site/abs/1806, 02952(2018).

[68] Chen C, Fragonara L Z. -05-21)[2019-06-30]. https:∥arxiv.gg363., site/abs/1905, 08705(2019).

[69] Yi L, Guibas L, Kim V G et al. A scalable active framework for region annotation in 3D shape collections[J]. ACM Transactions on Graphics, 35, 210(2016).

[70] Wang P S, Liu Y, Guo Y X et al. O-CNN: octree-based convolutional neural networks for 3D shape analysis[J]. ACM Transactions on Graphics (TOG), 36, 72(2017).

[71] Yi L, Su H, Guo X W et al. SyncSpecCNN: synchronized spectral CNN for 3D shape segmentation. [C]∥2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 21-26, 2017, Honolulu, HI, USA. New York: IEEE, 6584-6592(2017).

[72] Engelmann F, Kontogianni T, Hermans A et al. Exploring spatial context for 3D semantic segmentation of point clouds. [C]∥2017 IEEE International Conference on Computer Vision Workshops (ICCVW), October 22-29, 2017, Venice, Italy. New York: IEEE, 716-724(2017).

[73] Hackel T, Wegner J D, Remote Sensing, Spatial Information Sciences. III-, 3, 177-184(2016).

[74] Lawin F J, Danelljan M, Tosteberg P et al. Deep projective 3D semantic segmentation[M]. ∥Felsberg M, Heyden A, Krüger N. Computer analysis of images and patterns. Lecture notes in computer science. Cham: Springer, 10424, 95-107(2017).

[75] Wang W Y, Yu R, Huang Q G et al. SGPN: similarity group proposal network for 3D point cloud instance segmentation. [C]∥2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 18-23, 2018, Salt Lake City, UT, USA. New York: IEEE, 2569-2578(2018).

[76] Landrieu L, Simonovsky M. Large-scale point cloud semantic segmentation with superpoint graphs. [C]∥2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 18-23, 2018, Salt Lake City, UT, USA. New York: IEEE, 4558-4567(2018).

[77] Wang X L, Liu S, Shen X Y et al. Associatively segmenting instances and semantics in point clouds. [C]∥IEEE Conference on Computer Vision and Pattern Recognition, June 16-20, 2019, Long Beach, CA, USA. New York: IEEE, 4096-4105(2019).

[78] Li Z, Zhang J, Fang Z J et al. Road scene semantic segmentation with cycle-consistent adversarial networks[J]. Journal of Wuhan University(Natural Science Edition), 65, 303-308(2019).

[79] Xiong H J, Zheng X W, Ding Y L et al. Semantic segmentation of indoor 3D point cloud model based on 2D-3D semantic transfer[J]. Geomatics and Information Science of Wuhan University, 43, 2303-2309(2018).

[80] Zhang R. Research on polymorphic object semantic segmentation of complex 3D scenes based on laser point clouds[D]. Henan: PLA Strategic Support Force Information Engineering University(2018).