Single carrier 400 Gb/s transmission is the main application rate of next generation communication system. In order to improve the transmission efficiency of this rate communication in practical engineering application, the relationship between different modulation modes and transmission spectrum width in high-speed communication system is analyzed theoretically. Based on five different modulation formats of 16 order quadrature amplitude modulation(16QAM), 16QAM/32QAM, 32QAM, 32QAM/64QAM and 64QAM, the single wave input power, transmission capacity and transmission distance of 400 Gb/s transmission system are experimentally studied. The experimental results show that there are obvious differences in transmission capacity and transmission distance between mixed transmission mode and single transmission mode.