The nonlinear distortions of the erbium-doped fiber amplifier (EDFA) in the AM-CATV optical transmission system is analyzed and calculated. A new scheme compensating the nonlinearities of EDFA——optical compensation is proposed and the other two compensation methods are introduced. The results of the computer simulation and experiment indicate the distinct improvements for the nonlinear distortions of the EDFA using the optical compensation. At last, the three compensation methods are compared.