The main distortions of fisheye images of a field view 180° are the distortion due to the spherical structure and optical distortion. A step-wise calibration method based on the three-dimensional control field is proposed, in which the relationship between 3D space and imagery space is constructed with direct linear transformation (DLT), and the transformation radius associated with fisheye structure is calibrated with the iteration method combining the spherical perspective model. The distortion model is constructed based on the correction result of the first calibration, and introduced into DLT to calibrate the distortion parameters. The images are corrected again with the result of second calibration, which solves the serious distortion of fisheye images. The experiments verify the effectiveness and accuracy of the proposed method.