The measurement accuracy of the fringe projection three-dimensional system is mainly determined by the calibration accuracy of a camera and a projector. This study proposes an improved calibration method. Further, the projector is calibrated based on the inverse camera model. The phase coding method is employed to obtain the unwrapped phase, avoids the introduction of the camera calibration error, reduces the number of images projected by the projector, and simplifies and accelerates the calibration operation. Using two sets of fringe patterns, both horizontal and vertical, the projection by the projector corresponds with the camera image in case of system calibration. Thus, the pixel coordinate of the circular mark points in a projector image can be obtained. Subsequently, a radial distortion camera model is utilized for calibrating the projector. In addition, stereo calibration is the final step used to obtain a geometric relation between the camera and the projector. Furthermore, the experimental results demonstrate that the proposed calibration method is feasible.