In Brillouin optical time domain reflectometer (BOTDR) sensing system, it adopts optical coherent heterodyne detection to obtain spontaneous Brillouin scattering signal, making up for the lack of weak signal in direct detection. The system obtains the first-order optical wave of Brillouin frequency shift and the largest signal to noise ratio (SNR) with modulating LiNbO3 electro-optic intensity modulator by microwave source. But electro-optic intensity modulator is affected by many factors, resulting that SNR with the first-order optical wave of the reference light is low, which is heterodyne with the back Brillouin scattering light. Therefore, a new method of filtering reference light by fiber Bragg grating (FBG) is proposed, and the variation of SNR with the first-order side is analyzed. The experimental results show that the stable output of the first-order optical wave in a long time and Brillouin coherent signal with the maximum SNR can be obtained. The feasibility of the scheme is proved.