System calibration is a key procedure in fringe projection profilometry. The system calibration method of analogy binocular which regards projector as camera has the advantages of simple operation, high precision and high flexibility. Based on this method, a calibration method for accuracy enhancement of measurement system was presented. According to the extracted center coordinates of circle marks onto calibration board, the images for camera calibration which included the corresponding cross marks, were generated by a computer to avoid the influence of circle mark profile on corner extraction and improve camera calibration accuracy. In the meanwhile, the theoretical expression of fast calculating projector Gamma for phase correction was derived. Then pixel-to-pixel mapping between projector and camera can be accurately set up for projector calibration accuracy enhancement. Combined with the calibration results of projector and camera, it need establish unified world coordinate system for system calibration. The experiment results show by using the above calibration method, the calibration precisions of projector and camera can achieve 0.25 and 0.15 pixel, respectively, and the precision of a cuboid block measured can achieve 0.142 7 mm at the same time.