The background motion presented in the image sequence normally consists of complex motion patterns and is affected by local moving objects. In this paper, a new method to estimate the affine modal parameters based on even sampled pixel blocks in frames is presented. It leads to great reduction in computational load. In order to estimate the background motion using only the background pixels and reduce the influence of moving objects to a great extend, Huber function is considered to discriminate between the background pixels and the foreground pixels. Experiment results show that it is a fast and robust method.