To solve the problem of image matching failure under complex conditions including extreme non-rigid transformation, partial occlusion, and imbalanced illumination, a more robust template matching algorithm than the original best-buddies similarity (BBS) algorithm is presented. The similarity measurement between image patches is represented by Manhattan distance instead of Euclidean distance. On this basis, the new confidence map is constructed by sliding window to compute the BBS response values. The center of the brightest connected region is determined to be the last matching location after threshold filtering the confidence map by filtering process to eliminate the minor response values. Experimental and analysis results show that the proposed algorithm may be used to match the images with elastic deformation, similar region interference, partial occlusion, and extreme illumination change, etc.