This study proposes an improved spatial-temporal two-stream network to improve the pedestrianaction recognition accuracy of infrared sequences in complex backgrounds. First, a deep differential networkreplaces the temporal stream network to improve the representation ability and extraction efficiency ofspatio-temporal features. Then, the improved softmax loss function based on the decision-making levelfeature fusion mechanism is used to train the model, which can retain the spatio-temporal characteristics ofimages between different network frames to a greater extent and reflect the action category of pedestriansmore realistically. Simulation results show that the proposed improved network achieves 87% recognitionaccuracy on the self-built infrared dataset, and the computational efficiency is improved by 25%, which hasa high engineering application value.deep learning, spatial-temproal feature