Traditional gradient optical flow has large motion estimation deviation with real value, thus cannot be directly applied to image stabilization system. In order to improve traditional gradient optical flow, a pyramid multi-resolution coarse-to-fine search strategy was incorporated into this algorithm. Firstly, computing area affine transform parameters were selected as the final transform parameters. Then, in the compensation period, error control propagation method was selected to obtain long term stabilized sequence. The experiment results show the improved method can detect severe complex jitter, and can achieve the rotating precision less than 0.09°, translation precision less than 0.07, scaling precision less than 0.02, the compensation sequences' average PSNR raised 2.36 dB.