The fifth-order Kerr nonlinear coefficient of Λ-type three-level system is measured using cavity transmission spectroscopy. The fifth-order nonlinear coefficients for two-level and Λ-type three-level systems are numerically simulated. Comparing the results of numerical simulation, it is found that atomic coherence has an enhanced effect on nonlinear coefficients. Meanwhile, the simulation results of the third- and fifth-order polarizabilities are compared with the measured results, and it is found that the two results have good consistency.