In order to analyze the effect of optical material inhomogeneity on the image quality of optical systems with extremely small aberration, and guide the optimization of the optical systems, we propose a three-dimensional simulation and ray-tracing method. This method works in Zemax, using a self-defined program to build the three-dimension distribution of the material inhomogeneity. By setting the setp of the ray-tracing process and controlling the ray-tracing process, the simulation precision is enhanced. As an example, We study an optical system with numerical aperture (NA) of 0.7, wavelength λ=632.8 nm, wavefront aberration root-mean-square (RMS) value of 1.5 nm. The simulation result shows that the simulation precision of this three-dimensional simulation and ray-tracing method is obviously better than other two-dimensional methods.