Research on interactive behavior recognition has always been a research hotspot and difficulty in the field of machine vision research. For the problem of low recognition rate, this paper proposes a recognition algorithm that combines edge features of depth images, texture features of RGB (Red, Green, Blue) images, and optical flow motion trajectory features. First, Canny operator is used to extract the edge features of the depth images, local binary pattern operator is used to extract the texture features of the RGB images, and optical flow histogram is used to describe the dynamic characteristics of the images. Then, the extracted edge features and texture features are weighted and fused. Finally, static fusion feature and optical flow motion trajectory feature are coded and fused using the spatial pyramid matching model based on sparse representation to identify interactive behaviors. Experimental results based on MSR Action Pair, SBU Kinect interaction, and CAD-60 data sets show that the algorithm has a better recognition effect.