The calibration of the phase measuring deflectometry (PMD) system is an important factor affecting the accuracy. Aiming at the difficulty of the calibration of posture relationship between the screen and camera in PMD system, a new system calibration method is proposed. By constructing retro-reflectors on the screen, a photogrammetry system is introduced as the medium of transformation between the screen coordinate system and the camera coordinate system. Therefore, a high precision calibration of geometric parameters is achieved. The calibration accuracy is verified using a stereo PMD system. The experimental results show that the reconstruction surface flatness is less than 1 μm, which demonstrates that the proposed method is feasible and effective.