In view of the fact that most image restoration algorithms rely on fixed size sample blocks to search for the best matching blocks, ignoring the structure information of the sample blocks, resulting in the discontinuity and ringing phenomenon of the repaired images. Based on the approximation between the sample block and its neighborhood block, this paper designs an image restoration algorithm by using the structure information constraint rule and matching model. The information entropy feature of the image is introduced into the priority calculation of the blocks to be repaired, and the priority repaired blocks are obtained. Structural information measurement model is constructed by the number of known pixels in the sample block to measure the structural information of the sample block. Constraint rules of structural information are established according to the measured values to adaptively adjust the size of the sample block. The matching model is constructed by using the color and gray features of the image, in order to find the best matching block in the known region after adjusting the size of the sample block, and repair the block needing repairing. The experimental results show that the restored image obtained by the proposed algorithm has high structural similarity, good texture continuity, and no information discontinuity.