Due to the large distortion of fisheye camera images, the two-dimensional calibration board approaches are often difficult to obtain reliable corners around the board boundary, which may lower the calibration accuracy. Traditional three-dimensional calibration methods have some challenges on building a complex calibration field to obtain sufficient number of feature points. Therefore, a three-dimensional fisheye camera calibration method based on the laser scanner is proposed to achieve more accurate results. First, a laser scanner is used to obtain the point clouds of the fixed indoor environment. Second, an improved scale-invariant feature transform matching method is used to detect the corresponding features between point clouds and camera images. A two-round random sample consensus (RANSAC) algorithm is applied to rejecting outliers among the correspondences, and the internal and external parameters of fisheye camera are estimated based on the image and physical coordinates remaining corresponding points after the 3D RANSAC screening. Compared with the classical Zhang Zhengyou calibration method and its improved methods for various cameras, the proposed method can get more effective feature points to make calibration accuracy much higher, which can provide good lens distortion correction for fisheye camera. The proposed method is convenient, precise and extensively applicable.