The nonlinear effects that limit the performance of the multi-frequency probe (MFP) based coherent optical time domain reflectometry (C-OTDR) are investigated. Based on theoretical analysis and experimental results, compared with conventional C-OTDR, when the probe pulse has power gradient within the pulse width, self-phase modulation (SPM) and cross-phase modulation (XPM) are strengthened in the new C-OTDR scheme. The generation of four-wave mixing (FWM) is dependent on SPM and XPM, and with modulation frequency of phase modulator higher than 40 MHz, the stimulated Brillouin scattering (SBS) threshold can be enhanced by more than 5 dB, which benefits the maximum dynamic range of the MFP C-OTDR.