Aiming at the problems of high complexity, poor real-time performance, too many input parameters and the fusion model affected by the prior knowledge on existing infrared and visible image fusion algorithms, a Poisson fusion algorithm of infrared and visible images is proposed. In the algorithm, the expected gradient of the Poisson fused image is obtained by fusing the gradient of source images with the absolute maximum rule. The average fused image is decomposed into initial values and boundary conditions of the iteration. The Poisson fused image is reconstructed by solving the Poisson equation with the Jacobi iterative solution. For the two fusion needs of high quality performance and high real-time performance, optimal numbers of iterations are determined by experiments. In comparison with classical fusion algorithms, the proposed algorithm has batter performance for both quality-first fusion applications and speed-first fusion applications.