Focus on color fusion for infrared and visible images with low contrast between target and background issues, a contrast enhancement method for infrared and visible image was presented based on HSI color space. Firstly, the contrast of infrared and visible images was enhanced using local histogram equalization and median filter, then the infrared target was extracted from the enhanced infrared image after fuzzy threshold segmentation. Finally, the two enhanced images and the segmentation infrared target were fused into the three components of a HSI image in terms of a simple linear fusion strategy and color transfer. To enhance the color contrast between the target and the background, a scaling factor was introduced into the transferring equation in the H channel during color transfer process. The experimental esults show that, compared with other algorithms, the color fusion images of hot target and color contrast between low temperature object and background abtained by the presented method enhance obviously. At the same time, the details of visible images are endowed with natural color similar to that of the light color images during the day, which is more confortable to the human′s visual perception.