A method based on contour model tracking to estimate the homograhpy for the textureless object in the clutter scene is proposed. The initial estimation of the transformation is obtained in the framework of random sample consensus (RANSAC). The optimized homography solution is obtained by minimizing the normal distance. To calculate the initial transformation quickly and robustly, random three line segments conforming to the certain geometry constraint are utilized to solve the assumptive transformation relation. The transformation relation with the minimal errors are picked out as the homographic initial value. To overcome the issue that the mismatches of the model and image lines in the complicated background may lead to the failure of the homography optimization, in the process of matches of the model and image points, multiple image points matches are retained for each model sample point. In the weighting process for sample point, due to the use of the property of the sample point as well as the relation to the neighbor points, the robustness of method is enhanced effectively. Experimental results show that the proposed method can realize the optimized solution of homography in the complicated background. Contrast with the traditional method, the proposed method can overcome the influence induced by the complex background effectively and optimize the homography parameters for textureless objects successfully.