An algorithm of ray tracing based on light ray refraction and reflection in geometrical optics in the medium of variable refractive index is proposed and a recursive direction-cosine equation of reflected light and refractive light for ray tracing in the medium of variable refractive index is carried out, In the algorithm, some problems, confirmation of normal line, light turning etc., are resolved. At last its correctness is validated by comparing numerical value and analytic value of several different special distribution of refractive index of analytic solution, furthermore some problems in the during of applying the algorithm and under solution in the algorithm are discussed. Not only the algorithm has extensive feasibility and is independent of distribution of refractive index, but also its numeric precision is about 10-5. The application of the algorithm is prospective in the many cases, such as optical designing and spatial imaging compensation of variable refractive index.