On the issues about the robustness in visual object tracking, a novel visual tracking algorithm based on convolutional neural network (CNN) and conformal predictor (CP) is proposed. A two-input CNN model is constructed to extract the high level features from the sampled image patches and target template simultaneously, and the logistic regression is used to separate the object from the background. The CNN classifier is embedded into the CP framework, and the reliability of classification is evaluated via algorithms randomness testing. The classification result with credibility is obtained by region prediction at a specified significance level. The image patches with high credibility are selected as candidate objects, thus, the target trajectory is obtained through spacetime optimization. Experimental results show that the proposed algorithm can adapt to the occlusion, target appearance changes and complex background, and it has a better robustness and higher precision than the current algorithms.