The suppression of polarization crosstalk in PZT phase modulators as a key error source has been challenging for open-loop fiber optic gyroscopes (FOGs). We developed a polarization-diversity optical frequency domain reflectometry(OFDR) to measure distributed modulation polarization error in the modulator. The error contributes 8×10−6 rad to FOG’s bias instability. By using a UV-fabricated in-fiber λ/4 waveplate and polarization-mode converter with fiber taper technology, the modulation error has been suppressed by 15 dB in assembled FOGs. This approach reduced error with temperature from 25 °/h to 0.7 °/h, meeting the requirements of control-level gyroscopes with bias errors less than 1 °/h.