Deep convolutional neural networks are widely used in the image classification. Current convolutional neural networks architectures based on the simplified convolution can reduce the number of network parameters, but it will lose some of the important information, which decreases the performance of the networks. The two-stream convolutional unit is proposed, in order to improve the accuracy of image classification. The two-stream convolutional unit contains two convolutional filters, which extracts the features containing the information in and across the channels, respectively. Based on the proposed two-stream convolutional unit, a deep convolutional neural network called CTsNet is constructed. Experiments of image classification are conducted on the databases of CIFAR10 and CIFAR100. The experimental results demonstrate that the proposed two-stream convolutional unit can extract features containing the information in and across the channels separately, increase the diversity in features and reduce the information loss. The CTsNet based on the two-stream convolutional unit can improve the recognition performance effectively.