The orbital angular momentum (OAM) multiplexing and encoding technologies can effectively increase the channel capacity of the optical communication systems. In recent years, some researchers focus on using machine learning (ML) technology to detect OAM modes to improve the performance of OAM optical communication system. In this paper, the OAM modes detecting schemes based on ML technology are reviewed, including error back-propagating (BP) neural networks, self-organizing feature map (SOM), support vector machine (SVM), convolutional neural network (CNN), mode recognition techniques base on beam transformations and all-optics diffractive deep neural networks (D2NN). The performance, advantages and obstacles of each kind of the neural networks in atmosphere and underwater channels are analyzed.