In order to consider factors such as system aberrations into account in the nonlinear propagation simulation of high power laser, a ray-tracing method for nonlinear media is presented, and hybrid modeling for linear and nonlinear media is realized through programming. Three-lens focusing system with KDP crystal as the nonlinear media is taken as an example, and simulation analysis of the influence of nonlinear effect on beam quality is conducted with different laser power densities. The simulation results show that when the laser power density reaches 1.62×1010 W/cm2, the laser beam quality is degenerated evidently by nonlinear effect. Based on the simulation results, an optimization method, which takes the optical component intervals as compensation without changing the position of focal plane, is presented. The optimization results show that the focal spot size decreases by 2.8 μm, and the encircled energy under standardized dimension increases by 16.2%, thus the beam quality at focal plane is significantly improved.