An image representation method based on arbitrary shaped regions is proposed in this paper, and the motion estimation of the image sequence is executed according to this representation. The initial frame is smoothed while edge of the image is preserved. The smoothing algorithm is the modification version of Alvarez's method. The smoothed frame is segmented by the watershed method. According to the marker image, the image is stored in the form of region adjacency graph. To solve the problem of the over-segmentation, the criterions of region intensity and edge strength and region size are combined. The affine transformation is used as motion model for each region, and the nonlinear least squares method is used for the optimization. Compared with the method based on pixel, the result shows that the motion vectors produced by this method are more consistent and robust and the PSNR is improved.