A phase noise compensation algorithm, which carries out unscented Kalman filter (UKF) in time domain, is proposed based on high-order quadrature amplitude modulation (QAM) and large linewidth coherent optical orthogonal frequency division multiplexing (CO-OFDM) system. The algorithm realizes channel equalization based on the training symbol frequency domain Kalman filter at the receiver, and less frequency domain pilot data overhead is used to carry out frequency domain extended Kalman filter. The common phase error (CPE) noise is compensated first. Then, inter-carrier interference (ICI) phase noise coarse compensation is carried out on the data after CPE noise compensation in time domain. Initial pre-decision is performed on the frequency domain data after phase noise coarse compensation. Combining with the primary time domain data at the receiver, UKF is applied to the decision data in time domain to achieve fine compensation of ICI phase noise. Phase noise coarse compensation is performed for frequency domain data after fine compensation, and iterative operation is carried out, which greatly improves the compensation effect of phase noise. Simulations with transmission distance of 100 km are carried out based on 50 Gbit·s-1 CO-OFDM system. Compared with other algorithms, the proposed algorithm has higher spectral efficiency and better compensation effect. After two iterations of the transmission signal with laser linewidth of 700 kHz and modulation format of 32QAM, the bit error rate performance of proposed phase noise compensation algorithm can still reach the upper limit of forward error correction. The proposed algorithm promotes the application of large linewidth CO-OFDM systems in long distance access networks and metropolitan area networks.