Owing to imaging principle limitations, depth camera is impossible to obtain the true depth value of a non-Lambertian high reflective surface and transparent substances, resulting in model missing and reconstruction failure. Aiming at the problem of three-dimensional (3D) reconstruction on special surfaces, a method of positioning special surfaces and optimizing position tracking is proposed herein. In the proposed method, positioning for special surfaces is based on the statistics of the zero depth area and timing consistency constraint optimization. The confidence of the depth data is evaluated, and the non-uniform pose calculation method in space and time is used to reduce the influence of transparent materials on pose calculation. The experimental results show that in natural scenes, the proposed method can reconstruct and repair 3D models with special surfaces using only the depth data collected by a consumer camera, which makes pose tracking more accurate and increases model accuracy.