A dispersion flattened fiber (DFF) transmission system was proposed with high bitrate, polarization multiplexing and quadrature amplitude modulation(QAM) signal. The modulation format of the system was PM-16QAM, the bitrates were 160 Gbps and 256 Gbps. The transmission characteristics over DFF link system were experimentally studied, which were compared with those of non-zero dispersion shifted fiber (NZDSF) link and standard single mode fiber (SSMF) link. The experimental results show that the Error Vector Magnitude (EVM) of 160 Gbps and PM-16QAM over 50 km DFF link is 0.5% better than those over 50 km NZDSF link, the BER is 2 orders of magnitude better than that of NZDSF, and the DFF link can better reduce side-mode noise. When bitrate is 256 Gbps and transmission distance are 50 km and 75 km, the PM-16QAM signal can be well demodulated only after the DFF link. And the longer the transmission distance, the smaller the input optical power range is while maintaining good characteristics. Comparing 160 Gbps with 256 Gbps, high bitrate PM-16QAM signal transmission characteristics over the DFF link are superior to those over NZDSF and SSMF links. The higher the transmission rate and the longer the transmission distance, the more obvious the transmission effects are.