The dispersion effect of the Wollaston prism in the ultraviolet-visible polarization imaging spectrometer causes the center coordinates of the same spatial channel of the detector to shift and ultimately affects the accuracy of target signal detection. According to the polarization demodulation algorithm, spectral matching is also required when the modulation spectra of the two orthogonal components (S spectrum and P spectrum) exiting from the Wollaston prism are used to demodulate polarization information. In response, this paper proposes a spectral calibration and matching method. To start with, parallel light sources are used to calibrate the corresponding relationship between the field-of-view angle of the instrument and the pixel in the spatial dimension, and pixel coordinate collections corresponding to each spatial channel are extracted to determine a field-of-view calibration equation. Subsequently, in the same spatial channel, the corresponding relationship between wavelength and pixel is calibrated by the standard light source of a low-pressure mercury lamp to obtain a spectral calibration equation. Then, the results of field-of-view calibration and spectral calibration are utilized to match the orthogonal component spectra. Finally, the calibration results are tested on the characteristic wavelength of the Fraunhofer line in the solar spectra. The results show that the positions of the spectral absorption peaks of the two orthogonal components exiting from the ultraviolet-visible polarization imaging spectrometer have favorable consistency. Specifically, the deviation between the calibration value and the standard one is within 0.1 nm, which verifies the accuracy of the calibration results.