In recent years, deep learning techniques have achieved great breakthrough for its powerful non-linear computations in the tasks of target recognition and detection. Existing deep networks were almost designed based on the precondition that the visual data reside on the Euclidean space. However, many data in computer vision have rigorous geometry of manifolds, i.e., image sets can be represented as Grassmann manifolds. The deep network was devised based on the non-Euclidean structure of the manifold-valued data, which combined the differential geometry and deep learning methods theoretically. Furthermore, a deep network for image-set recognition based on the Grassmann manifold was proposed. In the training process, the model was updated by the use of the backpropagation algorithm derived from the matrix chain rule. Learning of the weights can be transformed as the Riemannian optimization problem on the Grassmannian. The experimental results show that this method not only improves the accuracy of recognition, but also accelerates the training and test process in one magnitude.