A new mixed kernel function weighted RX algorithm for anomaly detection in hyperspectral imagery was proposed. First, each spectral pixel was mapped into a high-dimensional feature space by a nonlinear mapping function. Second the nonlinear information between different spectral bands of the hyperspectral imagery was exploited with the RX algorithm in the feature space. In order to optimize the covariance matrix, each pixel in the covariance matrix was weighted according to its centroid distance. In this way the weighted covariance matrix could represent the background distribution better. Finally, the dot product computation in the high-dimensional feature space were converted into the kernel computation in the low dimensional input space. The new spectral kernel function and the radial basis kernel function were composited according to the characteristic of hyperspectral data to improve the performance of the proposed algorithm. To validate the effectiveness of the proposed algorithm, experiments were conducted on real hyperspectral data. The results show that the proposed method can detect more anomaly targets than the RX algorithm in the feature space.