The tracking by detection algorithm implements target tracking by detecting the target in each frames in real time.This algorithm aims to maintain an online training classifier,which intends to separate the target from the background for tracking by taking the samples from the background area as negative samples and from the target area as positive samples.But when the target was sheltered,or the shape of target changed in a large scale,how to sample and mark the samples accurately was critical for success tracking.A tracking by detection algorithm was proposed based on structured Support Vector Machine (SVM).Since the output of structured SVM can be very complex data structure,the position of the target was taken as the output of the structured SVM,which can overcomes tracking drift problem when the target was sheltered or the shape of target changed greatly.Experimental results show that the proposed algorithm has a good and stable tracking performance.