To deal with the problem that the camera tracking module of traditional visual simultaneous localization and mapping (vSLAM) can''t make pose estimation accurately, a semantic-based visual odometry is proposed. First, while using pyramid Lucas-Kanade optical flow to track and match the inter-frame feature points, the frame is pixel-wisely segmented. Then, the semantic information and geometric features are combined closely to accurately remove the outliers in the frame, thus the pose estimation and mapping can rely only on the trusted static feature points in the frame. Finally, a multi-scale random sample consensus (RANSAC) scheme is proposed. The matching points are sampled step by step, and different scale threshold are used for each step, which can reduce the detection time and improve the robustness of outliers simultaneously. Experimental results on the TUM data set show that, compared with ORB-SLAM2, the absolute trajectory error and relative pose error of the proposed system are improved by more than 90% in the high dynamic sequence. And the proposed scheme reduced the detection time by 30%-40% while the robustness of pose estimation is improved when compared with similar DS-SLAM.