In order to solve the calibration problem in augmented reality system, a novel optical transmission head up display system calibration method was proposed. First, the different coordinate systems are defined by analyzing the relationship among the components in system. Because of the distortion of camera and optical display system, virtual image distortion will be brought to some error. Therefore, the distortion of the camera and the optical display part is taken into account and corrected simultaneously when building the model. Finally, the non-linear regression estimation method is used to solve the model and calculate the calibration parameters. This calibration method, which combines optical display image distortion and camera imaging distortion correction, improves the calibration accuracy of optical transmission head up display system. The experimental results show that the average relative error of the calibration algorithm is less than 0.42% which basically satisfied the requirements of the system for calibration accuracy.