Diffraction beam deflection peak position of the acousto-optic device is proportional to the input signal frequency. It is the basic theory for acousto-optic spectrum analyzer frequency measurement. But the non-linear effects in diffraction causes diffraction peak deflection error. To get better measurement results, the diffraction intensity data should be processed to reduce the frequency of measurement variance. After the digital features of CCD background noise were analyzed, the background noise mathematical average had been taken as the threshold when denoising the data of diffraction intensity. According to the principle of least square method, fitting the diffraction intensity data by Gaussian function, the correlation coefficient is 0.997 6. Under the existing optical experimental platform, the measure variance is 542.5 kHz2 while the frequency was estimated by the diffraction peak position; the measure variance is 31.8 kHz2 while the frequency was estimated by the middle value of Gaussian fitting function. It provides theoretical support for reducing the frequency measurement variance of acousto-optic spectrum analyzer.