A new method to determine the optical constants and geometric thickness of single thin film is proposed with the measurement of the reflectance from spectrophotometer. Based on the Fresnel's optical expression, improved flexible tolerance method is employed in the case of a digital model of thin film by fitting the curve of measured reflectance spectrum. Simulation results show a satisfactory coherence of the optical constants and the thickness of the target film. By taking the influence of nonlinear condition into account with more direct and indirect limitation, the precision and value-searching efficiency have been improved. Furthermore, the problem of dimension degradation, which exists in "Downhill Simplex", has been successfully avoided. No initial input is needed for the procedure of optimization to achieve optical solution,which makes the whole processing of value calculation much more convenient and efficient.