In this study, we designed a repair algorithm to satisfy the requirement for repairing the defects in images. Based on the Criminisi algorithm, we introduced the marker-controlled watershed algorithm to segment an image into parts based on the scene for improving the image defect repair accuracy. Further, based on the nearest best matching principle, the image was repaired and a control function was introduced with respect to the pixel distance for reducing time for searching matching blocks. Then, we optimized the priority calculation method of the Criminisi algorithm by introducing the local color variance, and improved matching degree between the filled segments and the sections to be repaired based on the covariance. Subsequently, the confidence formula with respect to image repair was reset to reduce the magnifying error when the confidence value was continuously updated. Additionally, we considered many defect images as examples to verify the repairing effect, and the obtained results demonstrated that the designed algorithm has an excellent repair effect. Experimental results show that compared with other algorithms, the proposed algorithm can considerably reduce the time required to repair the damaged image and more effectively restore the original image information.