Based on introduction about principle of polarization three-dimensional image and components of four apertures polarization three-dimensional imager, the influence of four apertures relative polarization direction on polarization three-dimensional imaging accuracy was analyzed, to determine the accuracy requirements for four apertures relative polarization directions. The calibration method of relative polarization direction of four linear polarizers in four aperture lenses and calibration method of relative polarization direction of four apertures polarization three-dimensional imager was developed. The calibration device was built, and the relative polarization direction calibration was carried out in the laboratory. The calibration error of relative polarization directions of four linear polarizers in four aperture lenses was ±0.5o. The calibration error of relative polarization direction of four apertures polarization three-dimensional imager was ±1o. Thus, the normal direction error of three-dimensional polarization inversion can be guaranteed to be -5.47%~5.80%,that would ensure high-quality imaging of four apertures polarization three-dimensional imager.