During the analysis and interpretation of remote sensing image, it can greatly relieve the processing system of the heavy computing burden to extract targets of interest rapidly and correctly. According to the attention mechanism of human, a detection algorithm for targets of interest based on visual attention mechanism was proposed. Firstly a saliency map was computed by analyzing the spectral residual, and then salient target can be extracted without a priori knowledge by searching through the saliency map with a control mechanism of win-take-all competing and inhibition-of-return. A majority of salient regions were discarded by using the possible bound of size and shape for target of interest. Otherwise the focused regions were projected to the target feature subspace, and the difference in the projection coefficients can be used to determine whether the region is of interest or not. The proposed scheme is tested using an amount of high resolution remote sensing images, and the results show that it is effective.