In order to overcome the serious background interferences for small target detection of hyperspectral imagery, a nonlinear anomaly detection algorithm based on background residual error data was proposed. After the background endmembers were extracted, spectral unmixing technique was applied to all mixed spectral pixels to separate target information from complicated background clutter. Then, the unmixing residual error data that included abundant target information was mapped into a high-dimensional feature space by a nonlinear mapping function. Nonlinear information between the spectral bands of hyperspectral imagery was exploited and the anomaly targets could be detected by using RX operator in the feature space. Thus, the nonlinear statistical characteristics between the hyperspectral bands were used effectively on the basis of suppressing the large probability background information. Numerical experiments were conducted on real AVIRIS data to validate the effectiveness of the proposed algorithm. The detection results were compared with those detected by the classical RX algorithm and KRX which did not suppress the background information. The results show that the proposed algorithm has better detection performance, lower false alarm probability and lower computational complexity than other detection algorithms.