In terms of the low resolution of the depth image subjects to imaging noise interference acquired by time of flight camera, a second-order total generalized variation(TGV) depth image super-resolution reconstruction method based on the joint constraint of interpolated depth image and high resolution color image is proposed. Firstly, to obtain the gradient information of the interpolated depth image, the depth image with low-resolution is preprocessed by using the traditional interpolation and the multi-scale morphology methods. Then, the gradient information of the interpolated depth image and the high-resolution color image of the same scene are combined, and the regularization term in the second-order TGV model can be optimized as follows, the gradient information of the interpolated depth image is combined in calculating the anisotropic diffusion tensor. And to control the diffusion intensity during the reconstruction, a weighting factor determined by the gradient information of the interpolated depth image is introduced. Finally, the primal-dual algorithm is performed to complete the high-resolution reconstruction of the depth image. Experimental results show that the proposed method can effectively preserve the depth edge on the basis of the noise suppression, and a better high-resolution depth image can be achieved.