In distributed optical fiber sensing based on Brillouin scattering, Brillouin frequency shift (BFS) is linear to temperature and stain in the optical fiber. In order to improve the measurement accuracy of temperature and strain, an improved quadratic polynomial fitting algorithm is proposed. In the algorithm, the median filtering algorithm was proposed to preprocess the noisy Brillouin spectra, so as to improve the accuracy of gain peak location; then spectra within one linewidth and symmetrical about the peak gain were intercepted to extract the BFS precisely using the quadratic polynomial fitting algorithm. Firstly, after systematic comparison according to BFS error and error in frequency corresponding to peak value gain, the Brillouin gain of the same frequency corresponding to all spatial points was selected as the input signal. Subsequently, the effect of the proposed algorithm under different frequency intervals, signal to noise ratios (SNRs) and different filter window sizes was studied, meanwhile the optimal window size selection problem was investigated. The results show that the BFS error decreases first and then increases as the window size increases, and the general optimal window size ranges from 53 to 163.