The semi-global stereo matching (SGM) for remote sensing images is sensitive to noise and produces fringes in the areas with discontinuous disparity and weak texture, resulting in a low matching rate. An SGM algorithm for remote sensing images combined with speeded up robust features (SURF) is proposed herein. First, SURF is used to calculate feature-matching points and the main directions of the feature points in remote sensing images, and a fast nearest neighbor search algorithm is applied to eliminate the inaccurate matching points. Then, the Census transformation is used to calculate the matching cost of the two remote sensing images, and the path weight of the SGM algorithm in a different convergence path direction is adjusted by the main direction of the feature points. Finally, improved weighted joint bilateral filtering (WJBF) method is applied to refine the disparity to remove noise and fringes in the disparity maps. Experiments are performed on WorldView, IKONOS, and SuperView-1 remote sensing image datasets. Results show that the proposed algorithm is superior to the contrast algorithms in both subjective and objective evaluation indexes, effectively eliminating the fringes and noise in weak texture and disparity discontinuity area and improves the stereo-matching accuracy.