Optical communication systems are susceptible to various factors. Traditional methods for detecting abnormal data in optical communication systems have high error rates and detection efficiency. In order to obtain ideal abnormal data detection results in optical communication systems, a clustering analysis based feature extraction method for abnormal data detection in optical communication systems was designed. Firstly, a data transmission model for optical communication systems is designed, and clustering algorithms are used to extract abnormal data features. Then, a degree learning algorithm is used to establish an abnormal data detection model for optical communication systems, and genetic algorithms are used to optimize deep learning algorithms. Finally, a simulation experiment for abnormal data detection in optical communication systems is conducted, and the results show that the accuracy of the proposed method for detecting abnormal data in optical communication systems exceeds 98%, The detection time of abnormal data in the optical communication system is 21.6 ms, which has certain practical application value.