Based on the coordinate-transformation Orbital Angular Momentum (OAM) sorting scheme, this paper designs a phase-only FAN-OUT grating using a method based on iterative algorithm to copy beams and improve the resolution of the OAM beams sorting efficiently. This paper establishes a numerical simulation model for high-resolution sorting of OAM beams. Numerical results indicate that the energy in the spot received by the system with phase-only FAN-OUT grating is more concentrate and the minimum reduction of dispersion degree of the OAM spectrum is 0.35, the minimum reduction of the error probability is 15%. The phase-only FAN-OUT grating is advantageous for realizing high-resolution sorting of OAM beams.