Machine learning for integrated circuit hardware Trojan horse detection can effectively improve the detection rate. Unsupervised learning methods still have shortcomings in feature selection. At present, the research work mainly focuses on supervised learning methods. In this paper, the new characteristics of ring oscillator Trojan horse was introduced, and the hardware Trojan horse detection method based on unsupervised machine learning was studied. Firstly, the 5-Dimensional eigenvalues of each node were extracted for the circuit netlist to be tested. Then the local outlier factor of each node was calculated by LOF algorithm to screen out the hardware Trojan horse nodes. The simulation results of Trust-HUB reference circuit show that compared with the existing detection methods based on unsupervised learning, TPR (true positive rate), P (accuracy) and F (measurement) are improved by 16.19%, 10.79% and 15.56% respectively. The average TPR, TNR and A of hardware Trojan horse detection for Trust-HUB reference circuit reach 58.61%, 97.09% and 95.60% respectively.