Hand over face occlusion is a typical difficulty during depth sensor application. Aiming at this point, a gesture recognition algorithm is proposed using color and depth information comprehensively. Kernel fuzzy C-means algorithm is used to get rough segments of hand over face occlusion image and gray enhancement. The hand face separation is achieved. The initial level set function is introduced to solve the clustering method of gesture area pixels missing problem. The classification of gestures is carried based on the histogram of oriented gradient (HOG) features of depth map. The sample database is established by collecting different human hand and face close covered case gesture images. Comparative result proves the feasibility and effectiveness of the proposed algorithm. The proposed method can separate hand from face efficiently and get relatively ideal result, and the HOG features can describe the spatial information more precisely and get higher recognition rate compared with traditional characteristics.