To solve the problem of poor recognition effect of difference local directional pattern (DLDP), a face recognition method based on orthogonal gradient difference local directional pattern (OGDLDP) is presented. The pixel gray values in the fields of 3 pixel×3 pixel and 5 pixel×5 pixel are convolved with two different sets of 8 Kirsch operators, respectively. We differentiate the two sets of edge response values according to the corresponding numbers and take the absolute value in accordance with the corresponding number to obtain 8 edge response differences in horizontal and vertical directions. If the edge response values near the edge obtained in the 3 pixel×3 pixel field are made the differences before and after the counterclockwise direction and the absolute values are taken, the 8 edge response differences in horizontal and vertical directions will also be obtained. The OGDLDP code is produced by taking the maximum edge response difference values from the two orthogonal groups corresponding to the direction of the subscript, which are encoded into a double-digit octal number. The experimental results on Yale and AR face databases show that the proposed algorithm improves the recognition rate, and has robustness to the changes of illumination, expression, and shelter.