The Internet of Everything (IoE) era, envisioned by 6G technology, has brought about an increase of data collected from underwater objects and activities, which has in turn intensified the demand for more efficient and reliable underwater communication methods. Among the available options, underwater visible light communication (UVLC) stands out due to its superior speed, low latency, and enhanced security features in comparison to conventional acoustic and radio frequency communication methods. Consequently, UVLC has become a promising solution for future underwater wireless communication, attracting considerable attention worldwide^[1].

Among the many methods to achieve underwater wireless optical communication (UWOC), UVLC systems using blue-green LEDs have demonstrated potential for achieving high-speed and long-distance underwater transmission^[2-5]. However, one of the major challenges that restrict the performance of UVLC is the presence of nonlinear effects. These effects, caused by both linear and nonlinear components within the system, lead to signal distortions that greatly degrade the overall system performance. Among these, nonlinear effects, primarily resulting from photonic conversion at the receiver and transmitter ends, pose a more significant challenge compared to linear distortion. As the modulation order and rate continue to increase, there is a growing demand for higher signal quality within the UVLC system.

Traditional post-equalization methods such as decision feedback equalizers (DFEs)^[6] and the commonly used Volterra series^[7] have limited compensation abilities when it comes to addressing nonlinear distortions in UVLC systems. In recent years, neural networks (NNs) have demonstrated significant potential in capturing and approximating complex functions, leading to the adoption of AI algorithms in UVLC. These include LSTM-NN equalizers^[8], the time-frequency network TFDNet^[9], SVM-based post-equalizers^[10], and dynamic pre-equalization NNAEM^[11]. Despite their excellent equalization performance, AI equalizers face challenges such as high computational complexity, slower convergence, and poor generalization. However, the emerging approach of integrating prior knowledge of communication and physics into NNs can help overcome these limitations.

In this Letter, we propose a novel channel-estimation-based bandpass variable-order time-frequency network (CBV-TFNet). Given that nonlinear effects tend to be exposed and emphasized in the frequency domain, the integration of the time-frequency joint approach offers additional advantages to the model. In addition, a channel-estimation-generated mask is introduced before the network’s input layer. During the training process, we have innovatively designed a bandpass variable-order (BV) loss function to guide the neural network (NN) in effectively equalizing both out-of-band and in-band regions of the signal spectrum. The experimental results demonstrate that CBV-TFNet achieves a communication rate gain of 172 Mbps compared to DNN-based post-equalizers, while significantly reducing the time complexity. Furthermore, our proposed method achieves a remarkable bitrate of 4.95 Gbps over a 1.2 m underwater channel, using bit-power loading discrete multitone (DMT) modulation^[12]. To the best of our knowledge, this represents the highest bitrate achieved in UVLC systems utilizing a single-wavelength LED.

Figure 1 illustrates the architecture of the CBV-TFNet post-equalizer. Assume that the time-domain expression of the transmitted signal is $x(n)\in {\mathbb{R}}^N$ and the signal received is $y(n)\in {\mathbb{R}}^N$. In the proposed model, the short-time Fourier transform (STFT) is employed for feature extraction of the received signal, transforming the time-domain signal $y(n)$ into a two-dimensional (2-D) time-frequency spectrum $Y(\omega )\in {\mathbb{C}}^{M^{\prime }\times N^{\prime }}$. The STFT process unfolds as follows. First, the time-domain sequence undergoes sliding window processing. To avert spectral leakage caused by the marginal effect of the truncated time-domain sequence, a Hamming window of length $W$ is employed for this processing, with sliding window hop size $H$ continuing until all data points are processed. At each sampling time, an overlapping portion of length $O$ is retained from the data obtained by the sliding window, which satisfies the relation of $W=O+H$. Notably, when performing the inverse transform of STFT, the COLA constraint^[13] must be met to ensure lossless signal reduction in the conversion process, which can be satisfied when $O\ge W/2$. For the data procured by the sliding window, each window’s data are subjected to D points of discrete Fourier transform (DFT). Given the necessity for higher frequency resolution in post-equalization, the DFT points are generally considered as equivalent to $W$. The time-frequency signal can be expressed as $Y(\omega )=[Y_1(\omega ),Y_2(\omega ),\dots ,Y_N(\omega )]$. In the matrix, the $t$th column can be formulated as follows: $$Y_t(\omega )={\sum }_{n=tH}^{tH+W-1}y(n)w(n-tH)\exp [-j\omega (n-tH)/W],$$(1)where $w(·)$ represents the window function of length $W$. Given that the 2-D time-frequency signal obtained via DFT yields a complex signal, it is transformed into a real number prior to inputting into the DNN. The real and imaginary parts of each sliding window spectrum are integrated into a single column, setting the height and width of the 2-D signal at $2W$ and $\lfloor (N-W+H)/H\rfloor$, respectively. Finally, the 2-D spectrum is fed into NN.

The above process preprocesses time-domain waveform data in TFDNet^[9]. However, our experiments found that fitting DNN to frequency domain signals is slightly less effective than for time domain signals, potentially due to longer concatenated spectral data length. This reduces data processing capacity under the same NN. To mitigate this, transformations are introduced in two ways, as shown in Fig. 1. First, the pilot signal estimates the channel to generate a mask, preprocessing the spectrum sequence before network input, thus sharing the equalization burden and simplifying the network structure. Simultaneously, our proposed BV loss function guides the NN’s attention for spectrum equalization, allowing the NN to focus on the passband containing more effective information, resulting in improved performance compared to average allocation. For the mask generation component, transmitters generate a random sequence $x_P(n)\in {\mathbb{R}}^P$ as the pilot signal with a sampling rate identical to the transmitted signal. The LED transmits the pilot signal at the same working point, resulting in $y_P(n)\in {\mathbb{R}}^P$ after traversing the underwater channel. STFT is executed on both signals, eventually obtaining the channel transmission matrix $\text{HP}(\omega )\in {\mathbb{C}}^{M\prime \times P\prime }$ by dot division, $$\text{HP}(\omega )=X_P(\omega )/Y_P(\omega ),$$(2)and then aggregating it on the time dimension, $$M(\omega )=(1/P)·\sum _{t=1}^P{\text{HP}}_t(\omega ).$$(3)

Considering the equalization effect of NNs on nonlinear effect equalization, in order to avoid filtering out the out-of-band part of the signal spectrum, which is important to the nonlinear equalizer, by the mask derived from the inverse channel transmission matrix, we apply a smoothing operation on the resulting sequence, namely mean filtering: $$M_R(\omega )=\text{smooth}[M(\omega ),s]=\sum _{i=0}^{M^{\prime }}\frac{1}{s}\sum _{j=i-(s-1)/2}^{i+(s-1)/2}M(i).$$(4)

The STFT spectrum of the received signal is the dot produced with $M_R(\omega )$ before inputting into the network each time. Finally, the $t$th column of the 2-D spectrum can be expressed as $M_R(\omega )·Y_t(\omega )$. For the loss function part of the network, BV loss function is defined as follows: $$\mathrm{BV}(\mathrm{\Theta })=\frac{1}{T}\sum _{t=1}^T({\alpha }_{\text{Cut}}{\alpha }_{\text{Pass}}{\alpha }_{\text{Cut}})·\left(\begin{array}{c}{\sum }_{\omega =0}^{{\omega }_{\text{Start}}}\Vert {\widehat{Y}}_t(\omega )-X_t(\omega ){\Vert }_2^2\\ {\sum }_{\omega ={\omega }_{\text{Start}}}^{{\omega }_{\text{Stop}}}\Vert {\widehat{Y}}_t(\omega )-X_t(\omega ){\Vert }_2^2\\ {\sum }_{\omega ={\omega }_{\text{Stop}}}^{\omega }\Vert {\widehat{Y}}_t(\omega )-X_t(\omega ){\Vert }_2^2\end{array}\right).$$(5)

In this equation, ${\alpha }_{\text{Pass}}$ represents the weight of the model mismatch value within the signal passband, while ${\alpha }_{\text{Cut}}=1-{\alpha }_{\text{Pass}}$ signifies the weight outside the passband. The STFT spectrum outputted by the NN and the transmitted signal’s STFT spectrum acting as the label are divided into inside and outside passband frequency domain sequences, with the passband bandwidth $\mathrm{PB}=({\omega }_{\text{Stop}}-{\omega }_{\text{Start}})/2\pi$. By computing the L2 norms separately, the mismatch values for each part of the spectrum are obtained. Ultimately, the sum of these weighted mismatch values serves as the final loss.

It is worth mentioning that the bandwidth in the loss function, as a hyperparameter of the model, may not align with the theoretical bandwidth in communication systems. Due to the influence of nonlinear effects in the channel, a portion of the signal that was originally within the bandwidth might leak outside the passband. Therefore, when performing post-equalization at the receiver, it is important to consider the out-of-band information to reconstruct the original signal as accurately as possible.

Upon completion of network training, the prediction signal $\widehat{Y}(n,\omega )$ is outputted, and the predicted time domain signal $\widehat{x}(n)$ can be obtained by performing the inverse transform of STFT. Let ${\widehat{x}}_t(n)$ denote the inverse Fourier transform of ${\widehat{Y}}_t(\omega )$ when $tH\le n\le tH+W$: $${\widehat{x}}_t(n)=(1/N)\sum _{f=0}^{W-1}{\widehat{Y}}_t(\omega )\exp [j\omega (n-tH)/W].$$(6)

Subsequently, the obtained ${\widehat{x}}_t(n)$ undergoes windowing and overlap addition, followed by taking the ratio of the result with the overlapping addition of the squared window function. This yields the recovered time domain signal $\widehat{x}(n)$ as follows: $$\widehat{x}(n)=[\sum _t{\widehat{x}}_t(n)w(n-tH)]/[\sum _t{w}^2(n-tH)].$$(7)

Figure 2 presents the experimental setup of our UVLC system. The communication process comprises two stages: channel estimation and communication. First, QPSK modulated original data undergo DMT modulation and digital pre-equalization (DZN). The processed data are loaded onto an arbitrary waveform generator (AWG, Keysight M8190A), amplified through an electrical amplifier (EA, ZHL-6A-S+), and coupled with the LED’s driving current using a bias tee (ZFBT-4R2GW-FT+). The emitted signal passes through a 1.2 m water tank, lens, and aperture in the optical path. The received signal is captured by a photodetector (PD), producing a dual-output via a TIA, and fed into an oscilloscope (OSC, MSO9404A) for offline data processing. Synchronization, CBV-TFNet post-equalization, and DMT demodulation are performed for signal recovery. A bit-power loading algorithm optimizes modulation level allocation based on signal-to-noise ratio (SNR) calculations. In the next stage, this process is repeated with the newly generated modulation scheme until the desired bit error rate (BER) threshold for forward error correction (FEC) is achieved. Multiple tests confirm the viable communication rate.

The proposed CBV-TFNet employs critical hyperparameters, including the STFT window length, the number of hidden layers and nodes, and the passband bandwidth (PB) and weight of the BV loss function.

For proposed BV loss function, it is pivotal to adjust the bandwidth $W_P$ and weight ${\alpha }_{\text{Pass}}$ of the passband. Separate traversal of these at a working point data yields intriguing results by testing the communication system’s BER, as illustrated in Fig. 3. It is worth mentioning that this specific working point is a set of operating parameters selected by the benchmark method after traversing the working point, combined with theoretical derivation, with relatively strong nonlinear effects, but still near the optimal operating point.

The bandwidth of the generated transmitted signal is determined to be 775 MHz. In the general trend depicted in Fig. 3(a), the BER exhibits a decreasing trend with the increase of passband weight when values are less than 1, except bandwidth values of 723 MHz. This highlights the efficacy of the BV loss function in guiding the NN based on prior knowledge. However, the minimal BER is obtained at smaller weights when PB values are 723 MHz and 797 MHz due to nonlinear distortion causing leakage of the original band spectrum information. Consequently, equalizing in-band information while also giving some weight to out-of-band is crucial. The in-band to out-of-band information ratio differs across bandwidths, so do the weights corresponding to the minimal BER. As validation, the BER of 723 MHz at a bandpass weight of 1 is exceedingly high, and the equalized signal spectrum is illustrated in Fig. 3(b). When PB values are 797 MHz and the weight values are 0.9, the BER performance significantly improves, and the equalized spectrum is depicted in Fig. 3(c). It should be noted that the optimal hyperparameters of the loss function are related to the degree of nonlinearity of the operating point. Although they are not guaranteed to be the globally optimal parameters, considering the complexity of the work, the performance of the model is relatively robust and sufficient to support our work. Therefore, the values at this working point are chosen as the hyperparameter of BV loss function.

As for the mask added before NN input, it should be clarified that the mask proposed in this Letter is not intended to improve the model’s equalization performance per se, but to improve the convergence speed of model training. A drawback of CBV-TFNet and TFDNet in the previous work should be mentioned, that is, the training rounds are relatively long. With TFDNet, each training may require more than 700 rounds. By adding the mask, the number of training rounds can be significantly reduced. Figure 4(a) shows the error band plots for equalization performance with and without the mask or BV loss function model, across various training rounds. The results highlight the significant impact of the BV loss function on enhancing the system’s bit rate, which improves 100 MHz. Additionally, BV loss function narrows down the error band substantially, leading to improved model stability. Incorporating the mask further enhances the convergence speed of the model. The right subfigures in Fig. 4 depict the STFT spectrum of the received, transmitted, and mask-received signal.

The optimal STFT window length found through traversal and Bayesian optimization is used for NN parameters including baseline methods. The parameters and number of real multiplications are shown in Table 1. Upon setting hyperparameters, we apply different post-equalization methods to examine the system’s operation points using bit-power loading-DMT modulation. The bitrate is calculated and experimentally verified by the bit-power loading-DMT algorithm after assigning different subcarrier modulation orders according to the estimated SNR of different subcarriers, which ensure that the BER of the communication system transmission is below 7% FEC, that is, 3.8 × 10⁻³ threshold. Three post-equalizers’ rate contour plots are shown in Figs. 5(a)–5(c): Volterra-based, DNN-based, and TFDNet, reaching peak bitrates of 4.516 Gbps, 4.774 Gbps, and 4.855 Gbps, respectively. Figure 5(d) demonstrates CBV-TFNet’s superior nonlinear compensation performance, achieving a peak rate of 4.956 Gbps with the widest dynamic range.

The lower four graphs illustrate estimated SNR, QAM order, and power loading scheme when achieving maximum bitrates. Evidently, the CBV-TFNet equalizer improves high-frequency performance significantly, leading to higher bitrates. When using the number of real multiplications as an indicator for time complexity, CBV-TFNet outperforms the DNN equalizer, completing the same signal compensation task consuming only 38.15% of the multiplication times. This efficiency relies on the hop size of the STFT, with a longer hop size reducing the number of multiplications, without significantly sacrificing performance. In addition, using 4.75 Gbps as the threshold communication bitrate to quantify the dynamic range of each method, the data shown in Table 1 can be obtained. It can be seen that the proposed method is able to guarantee the highest dynamic range. Although the LMS-Volterra post-equalizer has smaller computational complexity, its performance in terms of peak bitrate and dynamic range is far from other methods.

It is worth noting that due to the large computational complexity of all the post-equalization methods we implemented, regarding their hardware parallel implementation poses a challenge. Regarding this problem, from an algorithmic point of view, we can use other optimization strategies to reduce the computational complexity, such as the pruning algorithm of NNs, which has been proven in our previous work to be able to drastically reduce the number of connections without significant degradation of MLP performance^[14]. Using knowledge distillation to compress the model can also achieve the effect of reducing the computational complexity of the model^[15].

We also encourage future researchers to further explore and optimize the post-equalization method in terms of hardware parallel implementation. By combining new algorithmic design and hardware optimization techniques, we can better address the issue of computational complexity and improve the feasibility and efficiency of the post-equalization method in hardware implementation.

Excellent performance of the proposed method can be attributed to the features of STFT, as reflected in Fig. 6. STFT enables the model to establish a multi-input-multi-output parallel structure, which outputs step-length data each time, thus reducing time complexity. Additionally, by offering a broader receptive field for the NN, STFT amplifies the model’s receptivity, with each data point passing through a network prediction process involving nine overlapping windows, yielding a receptive field of up to 136 data lengths—nearly double that of a DNN equalizer. Consequently, CBV-TFNet exhibits superior performance in learning nonlinear effects and inter-symbol crosstalk, displaying robust balancing capabilities.

In this Letter, we propose a post-equalizer for UVLC systems based on a time-frequency joint NN that incorporates channel estimation, CBV-TFNet. A novel BV loss function, also leveraging channel estimation, guides the NN’s focus on the spectrum within the passband that carries the majority of the information. Simultaneously, a pilot signal-based mask accelerates model convergence and streamlines the structure. Utilizing only 38.15% of real multiplication calculations compared to the DNN equalizer, the system achieves a bitrate of 4.956 Gbps in a 1.2 m UVLC system, which is the highest transmission rate for a single-wavelength LED used in the UVLC system as far as we know. Compared to the traditional LMS-Volterra post-equalizer, the proposed method achieves a bitrate gain of 440 Mbps and a significantly larger dynamic range. CBV-TFNet demonstrates a promising post-equalization scheme for free-space optical communication, including UVLC.