The target tracking algorithm is suffering from low accuracy and poor applicability due to the complex lighting conditions and severe changes in scale caused by the relative lens movement during panoramic video target tracking. To address this issue, we propose an algorithm for panoramic target tracking based on the improved SiameseRPN. First, the network structure of MobileNetV3 is used to extract the deep features to make the algorithm have a better adaptability to the scene defects appearing in panoramic video sequences, and the Squeeze and Excite module is used to increase the sensitivity of the network to feature selection. Then, we propose and construct a feature fusion module based on bilinear interpolation, which is used to make the output depth features of the last three layers have the same scale, and these three layers of features are fused for network prediction. Finally, we use a classification sequence to predict the positive and negative samples in the current sequence, and adopt a regression branch to predict the position information and scale information of current output targets. Thus the target position information is outputted. The experimental results show that the proposed algorithm has better tracking accuracy and it can effectively deal with the problems of poor local image quality and scale changes in panoramic data, while maintaining the real-time tracking performance. It shows a good adaptability to small targets, target occlusion, and multi-target cross movements in target tracking, and has good visual effects and high tracking scores.