Improving spectral efficiency via high-order quadrature amplitude modulation（QAM）formats for high-capacity transmission has been extensively investigated in coherent wavelength division multiplexing（WDM）transmission systems ^［1-7］. Stimulated Raman scattering（SRS）results in power transfer from short-wavelength channels to long-wavelength channels，which changes the signal power distribution across wavelengths along fiber propagation. The application of distributed Raman amplification（DRA）and，in general，the dependence of amplified spontaneous emission（ASE）noise on the wavelength results in a wavelength-dependent optical signal-noise-ratio（OSNR），where a longer wavelength channel has a larger OSNR. Therefore，a wavelength-dependent method is required to prevent the introduction of large system margins. Power allocation has become the method of choice to limit the imbalance caused by Raman amplification ^［8-15］. However，power optimization based on the analytical Gaussian noise（GN）model has high calculation complexity and limits the exploitation of the WDM system capacity to the maximum^［16-20］.

In this study，we propose an optimization method based on bit allocation that produces a higher exploitable capacity while being straightforward to implement. In the bit allocation optimization method，the QAM format per wavelength channel is identified using the received OSNR. Compared with power optimization based on a mathematical model，bit allocation optimization has very low complexity and does not require detailed knowledge of the characteristics of the network components，especially the parameters of the fiber and optical amplifier. For the first time，we designed a hybrid convolution neural network，long short-term memory network，and fully connected layer（CNN-LSTM-FC）structure for polarization division multiplexing（PDM）QAM signal equalization while incorporating a combination of the advantages attributed to the CNN，LSTM network，and FC layer structure ^［21-28］. In a C-band coherent WDM system covering a 4 THz bandwidth，we experimentally demonstrated an 80-channel 50 GHz grid transmission employing 48 Gbaud PDM QAM signals using a bit allocation method. Owing to the neural network（NN）equalizer，53.76 Tbit/s WDM transmission over 400 km standard single mode fiber（SSMF）has been achieved，satisfying the 25% soft-decision forward error correction（SD-FEC）threshold.

The experimental setup is shown in Fig. 1. Tx digital signal processing（DSP）comprises a pre-equalization and raised-cosine（RC）filter with 0.01 roll-off factor. Eighty WDM channels were produced at the transmitter，consisting of odd-channel（Ch. 1，3，5，…，79）and even-channel（Ch. 2，4，6，…，80）groups. We used 80 external cavity lasers（ECLs）with less than 100 kHz linewidth to generate WDM channels operating from 1531.51 nm to 1563.05 nm. The 40 channels in the odd-channel group corresponded to the H18-57 channels in the ITU-T standard with a 100 GHz frequency spacing，while the 40 channels in the even-channel group corresponded to the C18-57 ITU-T channels with a 100 GHz frequency spacing. Furthermore，two polarization-maintaining arrayed waveguide gratings（PM-AWG）were used to combine the odd and even channels.

The output electrical signals from the four independent channels（I_odd，Q_odd，I_even，and Q_even）of a 64 GSa/s sampling rate arbitrary waveform generator（AWG）were boosted and fed into two 30 GHz in-phase/quadrature（I/Q）modulators in the odd-channel and even-channel groups. After the I/Q modulator，a polarization multiplexer comprising a 3 dB polarization-maintaining optical coupler（PM-OC），1 m PM optical delay line，and polarization beam combiner（PBC）was used to generate PDM optical signals. Subsequently，an erbium-doped fiber amplifier（EDFA）was added to adjust the launch optical power into the fiber link. The fiber link consisted of four spans of 100 km SSMF with an attenuation coefficient of 0.188 dB/km. Each span had an ~18 dB ON-OFF gain backward-pumped Raman amplifier. Fig. 2 shows photographs of the experimental setup comprising 80 ECLs and a 4 × 100 km fiber link with Raman amplifiers.

The optical spectra of WDM signals employing the 256QAM format before and after fiber transmission at 0.02 nm resolution are illustrated in Fig. 3（a）and（b），respectively. SRS results in a non-flat optical spectrum after Raman amplification. The OSNR in a short-wavelength channel is low，and，unlike a long-wavelength channel，a short-wavelength channel cannot support high-order QAM formats. Therefore，to approach the maximum capacity of the WDM system，we adopted the bit allocation optimization method，where different channels carry QAM signals of different orders according to their OSNR.

After the fiber transmission，a tunable optical filter（TOF）was used to select the received optical signal of each test WDM channel. In addition，a variable optical attenuator（VOA）was added to adjust the input optical power into an integrated coherent receiver. We used another ECL signal as an optical local oscillator（LO）for homodyne detection. After optical-to-electrical conversion，the received PDM signals were captured by an 80 GSa/s sampling rate oscilloscope with a 36 GHz electrical bandwidth. In the offline Rx-side DSP，an NN equalizer based on the CNN-LSTM-FC structure was used after resampling，chromatic dispersion（CD）compensation，constant-modulus algorithm（CMA）equalization，frequency offset，and carrier phase estimation.

In the NN equalizer，we designed a hybrid CNN-LSTM-FC structure，as illustrated in Fig. 4. The front CNN layers extract high-order features of the input data，which compresses the input data ^［29］. The reported LSTM-FC layer structure has been widely verified to be effective in NN equalizers ^［30］. The proposed hybrid CNN-LSTM-FC layer structure combines the advantages of various NN layers. Therefore，the effectiveness of the NN equalizer is significantly higher than that of an NN equalizer consisting of single-type layers^［31］. Because the PDM QAM signals can be depicted by four real data points：I_x， Q_x， I_y，and Q_y，the input dimension of the network is T × 4，where T represents the memory depth of the network. After the input block，there are four cascaded one-dimensional convolution（1D Conv）layers in the CNN block. For simplicity，the parameters of the four 1D convolutional layers are the same. The tap of the convolution kernel was set to 1×7 to achieve its optimal performance. The number of feature maps constructed by the convolution layer was set to 24 to balance the performance and calculation complexity. The values of the convolution kernels were initialized using Gaussian weight initialization. The activation function of the CNN block was the ReLu function. At the end of the CNN block，a max-pooling layer was used to reduce the number of calculation parameters and eliminate information redundancy. The LSTM layer is a common model used in NN equalizers owing to its capability for storing time-domain information of the input. The hidden units of the two LSTM layers were set to 128 and 64. Subsequently，two fully connected layers，which contained 50 and 20 neural units，were used to output the I_x， Q_x， I_y，and Q_y values of the PDM-QAM signals. The nonlinear activation function of these two fully connected layers was a sigmoid function suitable for the output of the LSTM layer. A dropout layer was set after the first FC layer to suppress overfitting of the neural network. The loss function of the entire network was set as the mean squared error function. The batch size and training epochs were set to 64 and 40，respectively. The gradient optimizer was adapted from Adam.

First，we loaded the same data into the AWG in the transmitter and selected the optical signal of each wavelength channel using the TOF in the receiver. An optical spectrum analyzer was used to capture the selected optical signal and measure its OSNR. The received OSNR after fiber transmission versus the wavelength is shown in Fig. 5. Subsequently，in the bit allocation optimization method，the QAM format per wavelength channel was identified by the received OSNR. Therein，26 long-wavelength channels（1552.93-1563.05 nm）carried 256QAM signals，26 short-wavelength channels（1531.51-1541.35 nm）carried 64QAM signals，and the remaining 28 channels of the intermediate wavelength（1541.75-1552.52 nm）carried 128QAM signals. For each wavelength channel，we loaded the data with the corresponding QAM format into the AWG in the transmitter and selected the signal of this channel via the TOF in the receiver. The total line rate of this WDM transmission system was（26×6+28×7+26×8）×2×48=53.76 Tbit/s.

As shown in Fig. 6，after 400 km fiber transmission，the bit error rate（BER）of each channel is below the 25% SD-FEC threshold at 4.2 × 10^-2. Considering a 25% SD-FEC overhead，the net bit rate is thus 53.76 × 0.8 = 43.008 Tbit/s.

An 80-channel 50 GHz grid WDM transmission employing 48 Gbaud PDM signals was demonstrated using the bit allocation optimization method. Owing to the NN equalizer based on a hybrid CNN-LSTM-FC structure，53.76 Tbit/s line rate（43.008 Tbit/s net rate）transmission over a 400 km fiber link can be achieved.