Tracking by Detection (TBD) is a widely used framework in object tracking. Most TBD algorithms focus on object`s appearance model, but hard to consider both fps and success rate. Point to these problem, a new and rapid tracking framework is imported which uses the On-line Sequential Extreme Learning Machine(OS-ELM) to update object`s appearance model incrementally. Due to the learning speed of elm is fast enough, classifier could be updated every frame, so the classifier is more suitable to object`s apparent variations. The result shows this algorithm realizes real time tracking, and the success rate is higher than other TBD algorithms.