The principle of optical trapping is conventionally based on the interaction of optical fields with linear-induced polarizations. However, the optical force originating from the nonlinear polarization becomes significant when nonlinear optical nanoparticles are trapped by femtosecond laser pulses. Herein we develop the time-averaged optical forces on a nonlinear optical nanoparticle using high-repetition-rate femtosecond laser pulses, based on the linear and nonlinear polarization effects. We investigate the dependence of the optical forces on the magnitudes and signs of the refractive nonlinearities. It is found that the self-focusing effect enhances the trapping ability, whereas the self-defocusing effect leads to the splitting of the potential well at the focal plane and destabilizes the optical trap. Our results show good agreement with the reported experimental observations and provide theoretical support for capturing nonlinear optical particles.