Research and application of deep learning in recovery of three-dimensional structure from image or video, depth estimation, and real-time localization of visual sensor are introduced. Research progress of deep learning is overviewed. According to whether there is supervision, some representative deep learning algorithms and systems are introduced individually with deep analysis and comparison. Finally, the research spots on deep learning in recent years are discussed, conclusions are presented, and some research tendencies are discussed.