A phase noise compensation algorithm based on joint estimation of Kalman filtering (KF) in time and frequency domains is proposed aiming at higher-order quadrature amplitude modulation (QAM) based incoherent optical orthogonal frequency division multiplexing (CO-OFDM) system with large linewidth. Training symbols and pilot sequences in time domain are set in the transmitter. Frequency domain Kalman filtering is carried out based on training symbols at the receiver, and channel estimation value is obtained. Rough estimate value of phase noise is obtained with time domain extended Kalman filtering (EKF). The estimation value of phase noise in time domain is plural, and in order to obtain better compensation results, every OFDM symbol is divided into several sub-symbols according to the number of pilot sequences. Time domain EKF is carried out at the positions of pilot sequences. Rough estimation value of phase noise of every sampling point is obtained with linear interpolation at the last pilot of every sub-symbol. Predecision of frequency domain data after rough phase noise compensation and time domain phase noise fine compensation are carried out. Numerical simulation is performed by the proposed algorithm for CO-OFDM system with transmission rate of 50 Gbit·s-1 and transmission distance of 100 km. Results show that the compensation results achieved by the proposed method are better than that of other methods. When CO-OFDM system is with 16QAM and a laser linewidth of 1 MHz or with 32QAM and a laser linewidth of 800 kHz, bit error rate can reach the upper limit of forward error correction.