Three-dimensional structure reconstruction from image sequence is acquiring much attention in photogrammetry and computer vision. A new algorithm based on affine approximation is put forward to resolve the contradiction of accuracy and robustness in conventional algorithms. Firstly, it synthesizes those missing points in image sequence based on tri-linear tensor, and then rebuilds a new measurement matrix without missing elements, which is decomposed by the bilinear affine factorization method to recover the initial data for the three-dimensional structure of objects. Finally, the initial three-dimensional reconstruction is optimized by the “R-T”-model-based bundle adjustment. Synthetic and real image tests show the new method presents more accurate and robust result and it can be applied to aerial surveying and mapping, industry inspection, virtual reality, cartoon making, military reconnaissance, etc..