Integrating sphere and diffuser are always used as extended source, and they have different effects on radiance calibration of imaging spectrometer with parameter difference. In the present paper, a mathematical model based on the theory of radiative transfer and calibration principle is founded to calculate the irradiance and calibration coefficients on CCD, taking relatively poor uniformity lights-board calibration system for example. The effects of the nonuniformity on the calibration was analyzed, which makes up the correlation of calibration coefficient matrix under ideal and unideal situation. The results show that the nonuniformity makes the viewing angle and the position of the point of intersection of the optical axis and the diffuse reflection plate have relatively large effects on calibration, while the observing distance’s effect is small; under different viewing angles, a deviation value can be found that makes the calibration results closest to the desired results. So, the calibration error can be reduced by choosing appropriate deviation value.