When obtaining the real temperature by multi-spectral radiometric thermometry, the target emissivity information is the key to calculating the temperature. The general solution is to establish an emissivity model based on the function between emissivity and wavelength or temperature. However, when the assumed model deviates from the actual situation, it can cause large temperature measurement errors. Therefore, eliminating the interference of the unknown emissivity of the target, reducing the reliance on the emissivity model, and increasing the universality of the temperature measurement algorithm are the urgent challenges to be solved in multi-spectral radiometric thermometry. This paper propose an improved hybrid optimization algorithm, particle swarm optimization and genetic algorithm(HPSOGA). The core idea of the algorithm is to transform the multi-wavelength radiometric thermometry problem into a constrained optimization problem. Firstly, a group of spectral emissivity satisfying the constraint is initialized, constituting a population. The fitness value is calculated after taking the emissivity into the objective function established by the reference temperature model of multi-spectral radiometric thermometry. The population continuously evolves and iterates in the feasible domain by HPSOGA algorithm until the fitness value is the smallest. The corresponding temperature of each spectral channel is approximately equal. In this algorithm, the spectral emissivity and the real temperature of the target can be inverted simultaneously without assuming an emissivity model. Simulating six typical emissivity models verifies the new algorithms adaptability to the inversion of spectral emissivity with different distribution trends. The results show that the average relative error of the inversion temperature is less than 0.73% for the cases of true temperature 800 and 900 K. Finally, the algorithm is applied to process rocket motor plume flame temperature measurement data. The results show that when the design temperature is 2 490 K, the relative errors of the inverse temperature are less than 0.65%. Both simulation and experiment show that the new algorithm can solve the emissivity and true temperature to meet certain accuracy requirements. Therefore, the HPSOGA algorithm proposed in this paper is reliable and effective and provides a new way formulti-spectral radiometric thermometry to measure the true temperature of the target.