To enhance the optical computation's utilization efficiency, we develop an optimization method for optical convolution kernel in the optoelectronic hybrid convolution neural network (OHCNN). To comply with the actual calculation process, the convolution kernel is expanded from single-channel to two-channel, containing positive and negative weights. The Fashion-MNIST dataset is used to test the network architecture's accuracy, and the accuracy is improved by 7.5% with the optimized optical convolution kernel. The energy efficiency ratio (EER) of two-channel network is 46.7% higher than that of the single-channel network, and it is 2.53 times of that of traditional electronic products.