The quaternion spectral residual method for detecting the visual saliency regions of two images is proposed, by expressing the reference image and the distorted image as a pure quaternion matrix. Then both the method and the quaternion gradient features are employed to design color image quality evaluation, as well as visual saliency as the weight of the evaluation index. Numerical experiments are conducted on the TID2013 and CSIQ databases to calculate four kinds of objective evaluation indexes such as the Spearman rank correlation coefficient (SROCC), the Kendall rank correlation coefficient, the Pearson linear correlation coefficient, and the root mean squared error. The results show that the experimental SROCC value on TID2013 reaches 0.8169, which matches the subjective evaluation of humans.