Relative time delay of the optical device under test (DUT) is measured by optical low coherence interferometry by measuring the phase change of a broadband light after passing through it. The interference patterns of broadband light are recorded and analyzed using Fourier transform (FT) to calculate the phase change of the broadband light in frequency domain. The polynomial fitting algorithm is used to smooth the calculated phase curve. The derivation of phase delay curve versus frequency is the measured time delay. Measurement error of time delay comes from the intensity noise and pure phase noise of interference patterns. Theoretical analysis and simulation calculations show that the time delay error is directly proportional to phase error, and the phase error due to intensity noise is directly proportional to the noise too and it can be eliminated by polynomial fitting algorithm. Experimental results show that the instability of the phase characteristic of the DUT is mainly due to temperature variation, which is also the main source of measurement error. The relative time delay of a section of photonic crystal fiber (PCF) about 19 m from 1540 to 1560 nm is measured and the best precision reaches sub-picoseconds.