In order to improve the stability of the binocular stereo vision measuring accuracy image matching. A new robust matching algorithm based on feature point energy is proposed, the algorithm based on epipolar constraint to get the initial matching feature points around image. The energy not affected by affine transformation is defined to describe feature points based on epipolar constraint relationship among feature points, and the error matching is removed by comparing the energy of feature points. The proposed method effectively reduces the false matching rate and the false rejected rate so that it can meet the need of strictly limited false matching rate in the process of multiple angle measurement data. Experimental results show that the correct matching rate of the proposed algorithm is over 95%, and the false rejected rate is less than 2%.The algorithm has better robustness.