Confocal Raman microscopy can realise quantitative, nondestructive, and unlabeled imaging for sample microregions and obtain molecular structure characteristics. The time consumption of the confocal Raman microscopy system always lasts up to several hours or even tens of hours. During the time-consuming measurement process, it is highly vulnerable to some disturbing factors. These factors lead to the system's shift and make the sample out-of-focus, which results in poor imaging quality. This paper proposes a high-stability confocal Raman microscopy based on bilateral fitting subtraction to solve the above issues of the existing confocal Raman microscopy. Firstly, the proposed method utilizes the linear fittings about the data areas on both sides where the axial response of the confocal Raman spectral intensity curve is sensitive to the defocus of the sample position. The expressions of the two equations about the linear fittings are obtained. Secondly, through subtracting these two expressions, a differential expression can be obtained. The zero-crossing of the differential expression is the focal plane position of the confocal Raman microscopy. Monocrystalline silicon is used to test the focusing repetition accuracy of the proposed method. The monocrystalline silicon surface is axially scanned with a scanning pitch of 100 nm and the experiment is repeated 60 times. The results show that the maximum difference is 80.2 nm, which indicates that the proposed system exhibits a strong anti-drift ability. We also performed a Raman mapping imaging test on a standard atomic force step sample with a bar lattice of a period of 5 m. Compared with the image obtained by the confocal Raman microscopy without auto-focusing ability, the image obtained by the proposed confocal Raman microscopy is clearer, and the edge of the image is sharper as well, as a high signal-noise ratio. The simulation analysis and experimental results show that the proposed confocal Raman microscopy based on the bilateral fitting subtraction method can improve the focus ability and suppress the defocus effect caused by interference factors on the image quality. The proposed method provides a guarantee for obtaining a confocal Raman spectrum image with high stability and offers a new avenue of Raman spectral imaging method for automatic focusing and anti-drift.