An inverted structure of Y-branch LiNbO3 waveguide for measuring the multi-mode cut-off wavelength of the phase modulator is proposed and experimented. Analysis of the mode coupling shows that the output power of the inverted Y-branch waveguide is cyclical fluctuated with the wavelength when the waveguide is in multi-mode. And this characteristics can be used to judge the multi-mode cut-off wavelength. The validity of the method has been verified by simulation and experiment with sample.