Considering the influences of self-phase modulation and cross-phase modulation on the propagation characteristics of guided light in few-mode fibers, we present the analytic solution of the corresponding nonlinear coupled mode equation, which is used to calculate the nonlinear coefficients of few-mode fibers and analyze the nonlinear compensation in few-mode optical fiber communication systems. Taking four-phase phase shift keying (QPSK) signal systems which supports two linearly polarized (LP) modes as an example, the influences of mode loss and nonlinear effects on the mode division multiplexing system is analyzed by simulation, and the compensation effectiveness for inter-mode phase modulation and the adaptability of the compensation algorithm are also discussed in detail. Research results show that, for correct demodulation of QPSK signals after compensation, the maximum allowable misalignment of mode loss increases with the increasing reference value, and the nonlinear compensation effect in the few-mode optical fiber communication system is limited by the inter-mode nonlinear coefficient. The maximum allowable deviation of the inter-mode nonlinear coefficient is 0.146 W ^-1·km ^-1.