Crowd behavior recognition is an important research topic in computer vision field. Amid at the properties that the behavior of small scale crowd have the features both microcosmic and macroscopic, a small scale crowd recognition method based on causality network analysis is proposed. The trajectories of each pedestrians are calculated by covariance tracking to gain the nodes of crowd network. The Granger causality test is used to estimate the relationship between two pedestrians. Based on these causations, two types of complex network are generated which are pair-complex network and group-complex network. Some features of network such as the average path length, betweenness and clustering coefficient are extracted to recognize the six classifications crowd behavior (gather, chat, split, linger, meet and together). Experimental results show that the proposed method can express and recognize crowd behavior effectively.