In order to solve the shortcomings of the traditional Criminisi algorithm which the priority value tends to zero quickly and costs much inpainting time, an improved image inpainting algorithm is proposed based on information entropy and gradient factor. Firstly, the information entropy and the gradient factor for the image are fitted as weight factors, and the priority calculation method is optimized to find the optimal inpainting block. Secondly, the information entropy which can measure the complexity of the pixel block is used to adjust the search area of the matching block to establish a dynamic rule of the search area. Then, an adaptive model of the template size for the matching block is established with the help of the gradient factor to improve the optimal matching block search strategy. Finally, the sequential similarity detection algorithm is introduced to select the optimal matching block from the source region to achieve image inpainting. The experimental results show that compared with the traditional Criminisi algorithm, the proposed algorithm is effective both at the objective level and the subjective level. Moreover, the effectiveness of the image inpainting is more real, and the restored image has better visual effects.