According to the amplification theory of a few-mode erbium-doped fiber amplifier (FM-EDFA), the study on gain simulation is carried out for the four-mode multiplexed signal and the five-mode multiplexed signal. The genetic algorithm is used to optimize the three-layer erbium-doped four-mode group few-mode fiber amplifier and four-layer erbium-doped five-mode group few-mode fiber amplifier. The results of simulation and optimization show that the 1550 nm four-mode multiplexed signal is amplified by using the 980 nm two-mode multiplexing pump and the core pumped forward pump mode. The average gain of each mode is 24.48 dB, and the gain difference between the modes is 0.103 dB. The 980 nm three-mode multiplexing pump is used for amplifying signals, and the average gain of each mode is 23.31 dB. The gain difference between modes is 0.016 dB. By optimizing the pump mode combination and fiber doping structure, the gain performance of four-mode group and five-mode group few-mode fiber amplifiers in C-band is improved.