According to the accuracy and real-time performance requirements of image tracking system, a semi-physical simulation target tracking system was designed by using TMS320DM642 DSP as the core of control. The realization scheme of the function of both software and algorithm is presented in this paper. The hardware circuit is divided into three modules of video acquisition, video processing and peripheral interface circuit. Firstly, the target image's acquisition is completed by CMOS image sensor MT9V032, and the digital signal is transferred to DM642 through A/D converter unit. Meanwhile, the background differencing method and Mean Shift tracking algorithm are combined to determine the target location. The offset vectors are transferred to the servo system to drive the PTZ rotate synchronously for realizing accurate target tracking. The experimental results shows that the hardware design and algorithm can realize the real-time detection and tracking of moving target, and can provide feasible schemes for video tracking technology.