The depth map obtained directly is limited by the disadvantages such as low resolution and missing edge information, it greatly affects the application of depth map. In order to solve this problem, a two-channel convolutional neural network for depth map super-resolution is proposed. It consists of two channels, deep and shallow, and there are 21 layers in the deep network. Through joint convolution and deconvolution, combining skip connection and multi-scale theory, the deep channels can quickly learn the detailed features of depth map. Shallow network of 3 layers are used to learn the rough features of depth maps. Finally, the two channels are combined with details and outlines to realize end-to-end mapping from low resolution depth map to high resolution one. The model makes full use of the learning ability of the convolutional neural network to independently extract the effective features of the depth map and avoid the inaccuracy of manually extracting features. The experimental results on the Middlebury RGBD dataset show that the proposed model can achieve good results at a large sampling factor of 8, and has a high practical value.