Aiming at solving the problems that the existing stereo matching algorithms have, a low matching rate in noise intrusion and low disparity accuracy in discontinuous disparity and weak texture regions, a stereo matching algorithm based on fusion cost of Census transformation and mutual information (MI) and the segment optimization is proposed in this study. The proposed algorithm mainly involves two steps: initial disparity map acquisition and disparity map optimization. In the first step, the initial matching cost is formed by the fusion of MI and Census, and then, the cost is aggregated by improved guided filtering to obtain the optimal matching cost; the winner-take-all(WTA) strategy is used to obtain the initial disparity map. In the second step, the reference image is divided into superpixels, and a disparity plane is fitted to each superpixel; next, the average disparity of the superpixels is estimated using the Markov random field (MRF). Then, the average disparity is used to process the occlusion area in the adjacent system and optimize the disparity accuracy. Finally, the final disparity map is obtained by median filtering. The experimental results show that the average mismatch rate of the disparity maps of the 15 sets of Middlebury test datasets obtained using the proposed algorithm in a nonocclusion area is only 7.60%, running time of each stage is short, and average processing time for each pair of images is 6.8 s. Overall, the proposed algorithm runs efficiently.