A nonlinear equalization algorithm is proposed based on the general regression neural network (GRNN) in the coherent optical orthogonal frequency division multiplexing (CO-OFDM) system with high-order quadrature amplitude modulation and large laser linewidth. After phase recovery at the receiver, the training data is chosen to carry out the training and studying in the GRNN. In the process, the smoothing factor, the only parameter, can be decided in the GRNN. Then, for the detecting data at the receiver, the nonlinear equalization is performed by the GRNN. The numerical simulations have been completed by the proposed GRNN nonlinear equalization algorithm in the CO-OFDM system with a transmission rate of 50 Gb/s and a transmission distance of 100 km. Compared with the back propagation neural network nonlinear equalization (BPNN-NLE) algorithm, under lager laser linewidth and high-order quadrature amplitude modulation (QAM), the proposed method has a better nonlinear equalization performance and a shorter time of training running, which will greatly promote the application of CO-OFDM system in the fiber transmission with long and medium distance.