The Census transform is sensitive to noise, so it is difficult to obtain high matching accuracy with stereo matching algorithm. An anti-noise stereo matching algorithm based on the improved Census transform and outlier elimination is proposed. Firstly, at the initial match cost stage, the median of the window neighborhood is taken as the reference value and the outliers are controlled by the mapping function, which improves the reliability of the single pixel matching cost. At the cost aggregation stage, the outliers are eliminated from the initial cost value of dynamic aggregation window. Finally, the final disparity maps are obtained by disparity calculation and disparity optimization. The Middlebury benchmark images are used to test the stages of the initial matching cost, cost aggregation, and final disparity map on the VS2013 software platform. Experimental results show that the proposed algorithm has better noise-robust performance than the existing Census transform algorithms, and the error matching rate is 5.71%.