For the intelligent vehicle autonomous visual navigation need to get realtime, accurate, robust their own location information, a robust, efficient and accurate stereo visual localization algorithm is proposed. A new direct P3P pose estimation with RANSAC was proposed to obtain matching inliers and initial motion parameters and greatly improve accuracy. Then a new motion parameters optimization method was proposed based on sequence orthogonal iterative algorithm for general camera models. Recent N frames motion and structure parameters of stereo camera were rapidly optimized online. Compared with bundle adjustment, the speed and robust of the proposed method greatly enhanced, and the proposed method could attain global convergence. The outdoor real experiments show this method has high accuracy, good robustness and high computing speed in localization that can meet the high performance requirement of navigation localization.