In order to improve the detection rate， detection speed and scene adaptability of detection methods based on human visual system， a multi-scene infrared dim target dataset is constructed， and an infrared dim target detection algorithm based on visual attention mechanism is proposed. From the bottom-up perspective， the multi-scale gray and variance estimation is proposed to calculate the saliency map and estimate the optimal target size fastly. Then， the candidate targets are extracted using Features from Accelerated Segment Test（FAST） corner detection algorithm， and non-maximum suppression is introduced to reduce redundancy. From the top-down perspective， based on the theory of biological lateral inhibition and cosine similarity， a soft fuzzy adaptive resonance theory network is proposed， and a dim target feature set is constructed to train it. Finally， the candidate target is recognized by the well-trained model. The experimental results show that the proposed method has higher detection rate， faster detection speed and more stable performance in different scenes than five representative methods based on human visual system.