The wavelength calibration research on the automatic grating monochromator for orthogonal Czerny-turner structure was investigated. Combined with the structure parameters and characteristic of this monochromator, the sinusoid that accords with the grating equation was proposed as the function of the wavelength calibration. Based on the principle of the least square method, the formula of the fitting residual error of the wavelength calibration was given. Using the Nelder-Mead simplex method, the undetermined coefficient of the fitting residual error was solved, which founded the precise formula between the wavelength and the grating turning angle. The accuracy of the method was verified through the experiment. The calibrated wavelength precision of the monochromator is less than 0.1 nm, which satisfies the application requirement. Applied in the wavelength calibration of the automatic grating monochromator with orthogonal Czerny-Turner structure, this method is simple to apply and easy for implementation. Using this method, the wavelength calibration can be realized quickly and real-timely, but it is only needed to modify the control program of the stepping motor slightly, which shows a better practicability.