A compact binary feature for RGB-D face description and recognition is proposed. First, different from traditional hand-craft feature, we learned the compact binary feature from the training set using unsupervised learning method. Then, in order to make full use of the contextual information, we use the pixel difference vectors as the input. Finally, considering the smoothness of the depth image, we extract different size of pixel difference vectors from every block of RGB and depth image. This work demonstrates that the proposed method is highly discriminable and is robust to facial occlusion and illumination. And recognition rates are comparatively high on two publicly available RGB-D Kinect database.