The convolutional neural network structure fails to consider the independence and correlation between RGB images and depth images fully, so its detection is not high. A new double flow convolution network is proposed for the joint detection of RGB-D images. The RGB image and depth image are inputted to the two convolutional networks and the two networks have the same structure and weight sharing. After several convolutions, the independent features are extracted. According to the optimal weights in the convolution layer, the two convolutional networks are fused. The fused features are extracted continuously using convolution kernels, and the output is obtained by full connection layer finally. When the detection time is similar, the detection accuracy and the success rate are increased by 4.1% and 3.5% respectively, compared with the previous early and late fusion methods.