The color features were exploited in a novel framework for the unsupervised classification of urban areas in this paper. Firstly, based on the recent four-component decomposition model of the polarimetric synthetic aperture radar (PolSAR) data, the common color spaces, such as YUV, RGB, HSI, and CIELab were calculated. The color feature was quantitatively selected from these color spaces by introducing the color entropy. Then together with the texture feature and the extended scattering power entropy, the adaptive mean-shift algorithm was used to segment the PolSAR data into clusters. Finally, the clusters were merged according to the G0 distribution-based distance measurement. The proposed framework was verified by the experiments on one AIRSAR L-band and two Radarsat-2 C-band PolSAR data. The classification accuracy indicates that the proposed method has superior discriminative ability for urban areas compared with existing works.