The veiling glare index is a key criterion used to evaluate the image quality of optical systems. In general, the stray light coefficient can only be obtained with the actual measurement method after the optical system is processed. If results of actual measurement are unsatisfactory, then the method of suppressing stray light in the optical system must be adjusted to optimize its suppression effect. To overcome these questions, this study proposes a method for solving the veiling glare index during optical design. This method employs TracePro software to establish and analyze the optical system as well as trace rays. Matlab software is used to fit the simulation results and solve the veiling glare index. The proposed method can successfully calculate and analyze the veiling glare index in advance and accurately evaluate the stray light suppression of the optical system.