In order to solve the problem that energy image species (EIS) are susceptible to human movement time and position shift, i.e., it is difficult to describe the details of human behaviors, in this paper a method of human behavior recognition was present based on pyramid gradient histogram (PHOG) fusion features and a multi-class Adaboost classifier. This method first calculated the average motion energy image (AMEI) and the enhanced motion energy image (EMEI) of an objects silhouette images after human body registration, and then it extracted the PHOG features of AMEI and EMEI and series them together to form a kind of multi-level feature descriptor of human behavior. Finally, a look-up table-based real Adaboost (LUT-Real Adaboost) algorithm was utilized to realize human behavior recognition by designing a multi-class classifier. Experimental results show that the correct recognition rate in typical depth-included human action datasets is 97.6% by using this method, which is higher than that of other classifiers using single feature description and support vector machine. This reveals that, by combining global and local features, the proposed method can effectively describe the detailed active features of human behavior at different scales, enhance the description ability of human behavior characteristics, and improve recognition performance.