Three-dimensional fluorescence of 17 red tide algae species that belong to 13 genera of five divisions was measured by fluorescence excitation-emission matrix spectroscopy. And 2-D wavelet db7 was selected to decompose the spectra at different levels to choose the alternative characteristic spectra. Based on the norm reference spectra constructed by cluster analysis, the linear regression model was solved by particle swarm optimization (PSO) algorithm and the discrimination method was established at the division and genus level. Some samples were tested: for single algal samples, and the correct discrimination ratios (CDRs) were 96.1% and 73.6%, respectively; For simulative mixed algal samples, when the dominance were 60%, 75%, 80% and 90% of the total biomass, the CDRs of the dominant algae were 86.7%, 96.9%, 98.7% and 99.4% with the average relative contents of 62.6%, 72.7%, 76.0% and 81.6%, respectively at the division level. And the CDRs were 51.0%, 68.9%, 72.0%, and 78.8% at the genus level, respectively. For 364 actual mixture samples, the CDRs of the dominant species (75%) were 99.4% at the division level and 75.9% at the genus level . For the particular field samples from mesocosm experiment and corrected from Jiaozhou Bay, results showed that the method can be used to realize the identification of red tide algae population and estimate the relative abundance of different classes, especially between diatoms and dinoflagellates.