A decision-level fusion tracking method based on deep learning for infrared and visible spectra is proposed. By building the parameter transfer model, the visible detection model of the specified objects is extracted from the existing deep-learning-based detection model. This visible detection model is used as the infrared detection pre-training model, and the fine-tuning training on a collected infrared image dataset is done to obtain the infrared detection model based on deep learning. On this basis, a decision-level fusion tracking model based on deep learning is built. An comparison experiment between single-band tracking and dual-band fusion tracking is carried out. The research results show that the proposed method improves the tracking accuracy and success rate compared with the single-band tracking, and has good robustness.