Polarization aberration of optical systems in imaging polarimetry affects the polarization detection accuracy, especially in wide field of view and large relative aperture systems. The polarization aberration of the imaging lens in imaging polarimetry is demonstrated and analyzed through the way of polarization ray tracing. The impact of polarization aberration on the polarization detection accuracy of imaging polarimetry is also discussed. The variation of Stokes parameters as functions of the field of view and the relative aperture is achieved. The polarization aberration can be reduced and calibrated at different field of view and the relative aperture of the optical systems, and the correct polarization information of the object can be derived.