For the stereo matching method of deep learning based on patches, the network structure is vital for the calculation of the matching cost, and the time-consuming of convolutional neural network (CNN) in the image processing field also needs to be solved. We propose a stereo matching method of CNN based on a “shrink network”. The CNN method is utilized to train the similarity of the left and right image patches, and the matching cost of the stereo matching is obtained by the similarity. At the feature extraction stage, by adding batch normalization layers to each layer, the gradient dispersion in the backward propagation can be improved effectively. Besides, the full-connection layer adopts a "layer-by-layer reduction" form with other network optimizations to increase the speed while ensuring the accuracy. We utilize the KITTI datasets to test the algorithm. Experimental results demonstrate that the proposed method increases the accuracy and speed fairly compared to some other methods.