A deep learning network structure based on the convolutional neural network and long short term memory (LSTM) neural network is proposed. The feature fusion is used to extract the shallow features and deep features through the convolutional network, and the features are fused by convolution, and the the obtained vector information is input into the LSTM unit. Networks are trained separately using the optical flow images and the red green blue information, and the results from each network are fused with weights. The experimental results show that the proposed model effectively improves the accuracy of behavior recognition.