Using white diffuser to calibrate the imaging spectrometer has the advantages of easy realization and high calibration accuracy. For calibrating the imaging spectrometer with a white diffuser, the expression of the collected signal electron number of the imaging spectrometer detector elements is deduced to investigate the effects of the distance H of the standard lamp illuminating the white diffuser and the observation angle β of the imaging spectrometer on the accuracy of the imaging spectrometer calibration. The results indicate that bidirection reflectance distribution function (BRDF) has more influence on the observation angle than illuminating distance, whether considering BRDF or not. When H is in the range of 0.5~1.25 m, β is in the range of 0°~30°, and the relative electron number error is almost less than 1% whether considering BRDF or not.