Robust principle component analysis (RPCA) algorithm is introduced to eliminate the mass speckle noise in optical coherence tomography (OCT) system. We understand the characteristics of speckle noise in OCT system by analyzing the speckle generation mechanism in OCT system. Combining the characteristics of OCT system itself, the low-rank matrix recovered model based on RPCA algorithm is proved to be suitable for the speckle noise reduction in OCT system. The best estimation which decomposes the original image of OCT into speckle noise image and sample cross section image can be obtained based on the RPCA algorithm. RPCA algorithm can retain the speckle patterns of the sample’s own structure while separating the speckle noise, and avoid the generation of the artifact effectively. The result shows that RPCA algorithm can effectively suppress the speckle noise, enhance the signal-to-noise ratio, and improve the effect of OCT images, through comparing the images before and after processing.