In order to solve the problem that traditional two-frame and multi-view stereo matching have a poor effect and robustness in dealing with occlusion and low texture regions. A depth reconstruction algorithm based on epipolar plane image (EPI) is proposed to reconstruct complicated and fine scenes deeply. According to the special linear structure of EPI, a cross-detect model is proposed to detect the outlines of EPI, whose depth is computed by combining the exponent distance function and distance weight coefficient. The contour depth is used as a priori to the inner flat regions, and the priori likelihood is integrated into an energy function. The contour depth is propagated to the whole depth map by minimizing the energy function. The proposed algorithm is local, so it not only preserves the exact contour edge depth but also assures the smoothness of low texture regions. The test result shows that the proposed algorithm is superior to the primary in term of reconstruction speed and quality.