Here, a registration algorithm based on edge structure features is proposed to solve the difficulty of extracting feature points from infrared and visible images. First, the structural features of infrared images are enhanced using an optimized saliency algorithm. Second, we extract the stable edge structures of the infrared and visible images using a phase consistency algorithm. Further, the ORB feature points are extracted from the edge structures. Finally, the KNN algorithm and cosine similarity are combined to filter the matching feature points, and the random sample consensus (RANSAC) algorithm is used for purification. Experimental results show that the algorithm overcomes the influence of grayscale differences between infrared and visible images. In addition, it achieves a high registration accuracy and efficiency, which is conducive to the registration of infrared and visible images.