In order to overcome the problem caused by similar spectrum of the adjacent pixels, the clustering algorithms were introduced into the hyperspectral imagery abnormal target detection. A new algorithm using principal component quantization and density estimation on EM clustering was proposed in this paper. Using EM algorithm to cluster hyperspectral spectrum vectors in the high dimension space, the relations between adjacent pixels in spatial space were be represented by the relations inside or between classes. According to the theory that the abnormal target pixels would spread around the edge of the classes, abnormal target was detected in the unit of class to effectively avoid abnormal point information flooded. And this algorithm achieved good detection effect. For the requirement of EM algorithm initialization is sensitive, in related with the first principal component information of the imagery dataset, EM clustering algorithm was initialized by vector quantization and density estimation method. This can reduce the problems caused by initialization of EM clustering algorithm, and improve the detection effect of the algorithm and computation efficiency. With simulated and real AVIRIS hyperspectral dataset used in simulation experiment, the results show that the proposed anomaly detection algorithm is obviously superior to the traditional detection algorithm.