The RETE matching algorithm was a classical algorithm in the rule-based reasoning system, but when the number of rules and facts increased in the knowledge base, the generated intermediate match information greatly increased too, resulting to the large time complexity and space complexity, severely reduced the reasoning efficiency. To address this issue, this paper compared several improvement strategies of RETE algorithm, and optimization algorithm was proposed based on RETE cost model. The algorithm can automatically find the optimal RETE topology, reduce intermediate nodes, and greatly reduce RETE algorithm's time complexity and space complexity. The experiment shows that the running cost of optimized RETE algorithm is only about half the time than before optimization, and the reasoning efficiency is improved.