Wide field, high spectral resolution and high spatial resolution are the development directions for the imaging spectrometer. However, in order to extract the two-dimensional spectral information accurately, spectral and spatial distortion needs to be controlled within a small fraction of a pixel. The prism-grating imaging spectrometer is proposed. The mathematical model of prism-grating component is constructed by means of the vector method, and the optimization of structural parameters is determined through analyzing quantitatively the spectrum distortion characteristics of different prism-gratings. The optical system of ultraspectral imaging spectrometers with a direct vision is devised base on the Prism-Grating component. The spectral coverage, slit length and F number of the system are visible light from 400 nm to 800 nm, 14 mm and 2.4, respectively. The results demonstrate that the spectral resolution is 0.5 nm/pixel, and the modulation transfer function (MTF) for different spectral bands are more than 0.7 at the Nyquist spatial frequency of plane array charge coupled device (CCD). Both smile and keystone are less than 1 μm, and reduced to 13.5% of a pixel.