Two psychophysical experiments, experiment I and experiment II, were conducted to test color difference calculation method in images. Five ISO SCID images were used as test images and the test image pairs were displayed on EIZO CG 19 monitors. The computed color difference with CIELAB, CIEDE2000, CIE94 and CMC formulae showed that the calculated color difference in lightness and chroma attributes behaved very differently and the color image contents had some effects on the color sensation. An optimization method of color difference formulae is proposed based on the experiment I data. The optimized formulae CIELAB(1.501), CIEDE2000(2.291), CIE94(3.041) and CMC(3.381) perform much better than the original one and CIEDE2000(2.291) is the best among the four. The optimized formulae are also tested by the experiment II data, which proves that the optimized method proposed is accurate, simple and practical.