Mode coupling in few-mode fiber may cause the transmission performance deterioration of a mode division multiplexing (MDM) system, which is one of the major factors limiting MDM technology to be applied in practice on a large scale. Precise measurement of mode coupling coefficient in few-mode fiber and quantitative analysis on the relationship between mode coupling and the performance of MDM system can provide reliable basis for the system-damage compensation. The advantages and disadvantages of the present measuring methods for mode coupling in few-mode fiber are analyzed, and a measurement system of mode coupling in few-mode fiber with the structure of mode transverter/demultiplexer and an optical fiber circulator based on the Rayleigh backscattering theory is built. By compared with the measurement results of multiple-input-multiple-output power analysis, the measurement performance of the proposed system has been verified. The experimental results show that the proposed system can realize the measurement of the mode coupling distribution in few-mode fiber with the fiber length of 9.8 km, and the measurement results are stable.