A new target detection algorithm in hyperspectral imagery based on endmember extraction method is introduced. The purpose of our target detection is to locate and search for targets which are relatively small with low probabilities in an image scene. We use the transformation matrix of principal components analysis to construct an orthogonal subspace projection operator that projects the hyperspectral image onto a subspace, which is perpendicular to the space spanned by transformation matrix. In this subspace, background information is effectively suppressed and small targets become obvious. So, endmember spectra of targets can be extracted using iterative error analysis method. Then, we segment the targets according to the spectral angle between selected endmember spectra and each pixel vector of hyperspectral image. The proposed algorithm was studied using real hyperspectral data and compared with RX algorithm. Experimental results show that the algorithm can effectively and reliably detect the small target without prior knowledge.