The background difference method based on Kalman filtering cannot adapt to the background update and it is sensitive to light changes and object moving. A modified background subtraction algorithm based on the idea of classification is proposed. First, the initial background model is gotten by averaging the first N frames of the video sequence images. Then, the difference image is obtained from the difference between the Kth image and the background image. The difference image is split into foreground and background blocks for two times and the classification criteria are the mean value and standard deviation. The foreground blocks are finely segmented based on a single pixel, and the moving targets region is determined. Finally, the adaptive background updating is conducted according to the gray value information between adjacent frames. The experimental results show that the proposed algorithm can effectively solve the problems of slow changes in external light and slight movement of objects in the background, and it has good robustness, relatively higher computing speed, and accurate moving targets area.