The CSK algorithm was used to extract a least square classification of moving objects from image fragments in this paper, and the multichannel color features was introduced to calibrate the moving objects. Through the cyclic hypothesis of periodicity of the kernel function in the current image fragments, the CSK algorithm was applied to compensate the lack of target gray-level features describing capacity with CSK algorithm in some extent. The PCA was used to reduce the feature dimension, remove feature redundant information, improve the updating speed of classifier parameters. The problem of moving target tracking could be solved when CSK algorithm classifier parameters were updated linearly and could not adapt to large changes of target. Experiments were performed on the algorithm dataset of the benchmark test platform and the dataset of test data. The experimental results of target color kernel tracking algorithm (TCKCT) show that the algorithm has a better tracking effect in the case of the illumination changing, the background clutter, the target deformation existing, the target moving velocity is faster and the target motion amplitude is larger. The experimental results of UAV tracking remote control car further verify the characteristics of TCKCT algorithm and good real-time performance can meet the target tracking requirements of UAV. It has a good practical application prospect.