Two comprehensive evaluation metrics, image perceptual quality based on target detectability (PQTD) and perceptual quality based on scene understanding (PQSU), are proposed to measure image quality for visible and infrared color fusion images of typical scenes. A psychophysical experiment is performed to explore the relationship between conventional quality attributes and the proposed evaluation metrics. The prediction models for PQTD and PQSU are derived by multiple linear regression statistical analyses. Results show that the variation of PQTD can be predicted by sharpness and perceptual contrast between the target and background, and that color harmony and sharpness can predict PQSU. The proposed evaluation metrics and their prediction models provide a foundation for further developing objective quality evaluation of color fusion images.