In order to research the relation between self-emission spectrum and transmittance of target at oxygen A-band, the method for fitting the baseline of oxygen A-band spectrum based on the partial least square regression (PLSR) is proposed according to the transmittance calculation method. The experimental system is set up to verify the accuracy of baseline fitting. Firstly, the calculation method of oxygen A-band average transmittance is given according to the blackbody radiation theory. The spectrum of the target to be measured is used as study object. The PLSR is used to fit the baseline of the target to be measured at oxygen A-band according to the out-of-band data. For the sake of improving the fitting accuracy, the singular point of measurement is eliminated. The baseline fitting uncertainty is used to evaluate the fitting accuracy of PLSR. In order to verify the accuracy of this method, the spectra with different distances and different resolutions at the same distance is measured in the range of ~130 m using the halogen light. These spectra are fitted and each standard deviation is analyzed. The results show that the average standard deviation is 0.23% with different resolutions at the same distance. The baseline fitting uncertainty becomes smaller and signal to noise ratio is increased with the resolution reducing. In addition, the baseline fitting uncertainty is related to the resolution of the test equipment. The higher the resolution is, the larger the uncertainty of out-of-band data baseline fitting becomes, whereas, the smaller the out-of-band baseline fitting becomes.