The stereo matching algorithm based on convolutional neural network improves the accuracy considerably; however, most algorithms still cannot meet the real-time requirements. In this study, we propose a real-time stereo matching algorithm with hierarchical refinement, which initializes the disparity map at a low-resolution level and gradually restores the spatial resolution of the disparity map. The proposed algorithm uses a lightweight backbone network for extracting the multi-scale features and simultaneously, the features are inversely fused to achieve an improved robustness without significantly affecting its real-time performance. Furthermore, we propose a multi-branch fusion module to progressively refine the disparity map. After the different modes in different regions are automatically clustered, the residuals of the disparity map are predicted. Subsequently, the final results are combined based on the cluster weights to ensure that the regions exhibiting different characteristics can be effectively processed. Based on the KITTI test dataset, the operating rate obtained using the proposed algorithm is 20 frame/s, and the error rate is reduced by approximately 30% when compared with the DispNetC algorithm. These exhibit a comparable operating efficiency.