In order to resolve the current problems of fundamental matrix estimation caused by lacking of obvious feature points, according to the epipolar geometry of the horizontal lines and vertical lines, a new method of calculating the fundamental matrix that can be combined with these two kinds of line features is proposed. Firstly, camera orientation elements and relative attitude are used as the elements of fundamental matrix, the equivalent relationship based on horizontal lines and vertical lines is deduced, and then the line features are made as the interior point of the RANSAC algorithm, after that the weight of each factor is determined combined with M-estimators, and finally the unified adjustment model is built. Experimental results of simulation data and real images show that the proposed approach is feasible in practice, and can reduce the dependence of intraditional method on feature points, and the introduction of the line features can improve the accuracy and stability of the calculation results to a certain extent.