In the water with petroleum pollution, the petroleum will be adsorbed on the surface of suspended particulate matter and form a double-layer structure, which impacts on the spectrum characteristics to the scattering coefficient. It is a key to improve the accuracy of the scattering theory model that the contribution of petroleum substances and suspended particulate matter on the scattering spectrum coefficient is be separated. A backward scattering coefficient spectrum measurement system was being built from linkage observation of three kinds of instruments, including DAWN HELEOS Ⅱ18 angle scattering measuring instrument (Wyatt company, American), LISST-100-xB size instrument(SEQUOIA SCIENTIFIC, INC, American), HydroScat-6 Sprctral Backscattering Sensor (HS6) ( Hobilabs company, American). Many parameters were measured such as voltage value of the scattering intensity, the particle size distribution, particle concentration and backward scattering coefficient in different water samples. Using the Mie scattering theory, a new algorithm to separate the scattering coefficient spectrum and new way of thinking to calculate volume scattering function β(λ, θ) of the unknown refractive index material were proposed. The matching experiments were done by selecting quartz sand as particles whose refractive index (m) is known and petroleum sewage collected from different oilfield area. On the basis of matching experiments different water samples with different properties were obtained and related data were determinated. Firstly, according to Mie scattering theory the water volume scattering function β(λ, θ) for each sample is calculated. Secondly, the equation was set up which can convert the scattering intensity corresponding to the voltage value V(θ) measured by DAWN HELEOS Ⅱ 18 Angle laser scattering instrument into volume scattering function β(λ, θ). Thirdly, according to the method of optimum the equivalent refractive index (mos) of the oil sands mixed and the refractive index (mo) of petroleum sewage were estimated; Finally, using β(λ, θ) and estimation of mos values and mo values to calculate the backscatter coefficient bb(λ) of all kinds samples, and new algorithms were set up which seperated quartz sand bb, s(λ) and petroleum sewage bb, o(λ) from mixed water with petroleum and sands respectively. The establishment of these separation algorithms improves the accuracy of the scattering theory model of the water petroleum pollution, on the other hand expands the Mie scattering theory in the application of ocean color remote sensing.