In order to evaluate the existing color harmony models, 24 color centers uniformly distributed on the CIELAB color space are selected and the observers with normal color vision are organized to assess the color harmony of patches and garments scene. The results of color harmony obtained from the psychophysical experiment are used to test the existing color harmony principles and models. The results show that color harmony scores obtained from the patches and garments have a weak correlation coefficient (R2<0.126). Predicting results by existing color harmony models are unsatisfactory with experimental data (R2<0.5). Therefore a new patches color harmony model based on Ou model and a new garments color harmony model based on Luo&Ou model are optimized, and the performances are considerably improved in different application scenarios. The experimental results can provide a study method and reference for the prediction of color harmony.