The curved-prism-based hyperspectral imaging technology has become a research hotspot in recent years. However, the non-coaxial characteristic of two spheres in a curved prism makes the fabricating procedure of a curved prism far harder than that in a traditional coaxial optical system. Fabricating error is the key factor influencing the final imaging quality of a spectrometer. At present, the modulation transfer function (MTF) is mostly regarded as an important evaluating indicator, however it does not consider the effect of fabricating error on spectral distortions including spectral smile and keystone. In this paper, the generation mechanism of spectral smile and keystone of curved prisms is investigated with the geometric-optical method. In addition, a mathematical model is proposed for deriving the relationship between fabricating errors of curved prisms and spectral distortions including spectral smile and keystone, and the effect of fabricating error on the spectral distortions of the hyperspectral imaging spectrometer is analyzed. Moreover, through geometric ray tracing, the analysis results of fabricating errors of curved prisms are verified. The results show that there exists an obvious difference in the sensitivity of spectral smile, keystone, and MTF to fabricating error. In order to ensure that the drop limit of the system MTF induced by fabricating errors of curved prisms is smaller than 10% of the design value, the tolerance for the second allocation is conducted to obtain the final tolerance allocation result, in which the fabricating tolerance with the minimum tolerance limit value is a tilt error at X-axis. The research here provides a reference for the actual fabricate procedure.