A method based on phase modulator is presented to measure retardation of the quarter-wave plate by swinging it without the information of its fast axis direction. In this method, collimated laser beam goes through a polarizer, a phase modulator, the quarter-wave plate to be measured, and an analyzer, and finally incident upon the detector. Its electric signal is amplified and filtered for data processing. The retardation can be attained by swinging the quarter-wave plate, without the precondition of the knowledge of fast axis direction. In experiments, a quartz quarter-wave plate is measured, and its repetitiveness is 0.13° and the reproducibility is 0.17°. Our result is consistent with the one got by the method on the photoelastic modulator when the azimuth of the fast axis is 0°.