In this paper, recent advances in underwater wireless optical communication (UWOC) are reviewed for both LED- and LD-based systems, mainly from a perspective of advanced modulation formats. Volterra series-based nonlinear equalizers, which can effectively counteract the nonlinear impairments induced by the UWOC system components, are discussed and experimentally demonstrated. Both the effectiveness and robustness of the proposed Volterra nonlinear equalizer in UWOC systems under different water turbidities are validated. To further approach the Shannon capacity limit of the UWOC system, the probabilistic constellation shaping technique is introduced, which can overcome the inherent gap between a conventional regular quadrature amplitude modulation (QAM) format and the Shannon capacity of the channel. The experimental results have shown a significant system capacity improvement compared to the cases using a regular QAM.