Because the traditional kernel correlation filter algorithm for visual object tracking has low tracking accuracy under fast motion, background clutter, and motion blurring conditions and cannot deal with scale changes, a real-time object tracking algorithm based on context awareness and scale adaptation is proposed. Based on the kernel correlation filter algorithm framework, context-aware and scale-adaptive methods are introduced to add background information and handle changes in the scale of the target. First, the target region is sampled using the features of the fusion histogram of oriented gradient (fHOG), color names (CN), and gray, and a two-dimensional translation filter is trained. Then, a scale pyramid is established in the target area and multi-scale sampling is performed using fHOG on the target area. Following this, a one-dimensional scale filter is trained. Finally, the update strategy is improved in the model updating stage. The experimental results of 100 sets of video sequences in the standard OTB-2015 dataset show that the proposed algorithm showed an improvement in the accuracy by 13.9% as compared with the benchmark algorithm (kernel correlation filter, KCF), and the success rate improved by 14.2%, which is superior to that of other comparison-tracking algorithms considered in the experiment. Under the conditions of scale change, motion blur, and fast motion, the proposed algorithm can maintain a high speed with accurate tracking.