The visual target tracking algorithm based on deep learning spectrum convolutional neural networks was presented. The spectral pooling was adopted instead of max pooling in the deep convolutional neural network, then the softmax loss layer was replaced with Bayesian theorem to compute maximum classifier score, and integrated it into the deep neural network tracking framework. The location of the target can be obtained by calculating the probability distribution of the input samples. The advantages of feature dimension reduction at random with spectral pooling and computation efficiency was taken to avoid much spatial information lost, which also helped to improve the computation speed. Compared with the original algorithm and other state-of-the-art methods, the proposed tracking method shows excellent performances on test baseline dataset.