Phase measuring deflectometry (PMD) based on fringe reflection has been widely studied as a way of obtaining three-dimensional shape of specular objects. System calibration is an important step, and it determines the accuracy of the measurement results. We propose a calibration method to obtain the system parameters based on phase information. As a result, it can build the relationship between the absolute phase map and depth data. A contrast experiment is done for verification about extrinsic parameters of the LCD screen by phase data and the checkerboard. The experiment shows that the method using phase data is more accurate when images are out of focus. Using the calibration system, we test a concave mirror and an artificial specular step with discontinuous reflective surface, and the error is about 22 μm. Experiment results show that the proposed method can precisely determine the system parameters, so that 3D shape of specular objects can be measured with a high accuracy.