For the purpose of scanning and measuring 3D surfaces efficiently, a non-contact optical coordinate measuring system was established by installing a point laser sensor on the mobile terminal of Z axis of a CMM through a probe head. As a result of converting one dimensional distance values of the laser sensor to three dimensional coordinates of the measuring points on the surface, a calibration method of the space vector of laser beam based on a spherical surface was proposed. In the calibration procedure, the trajectory of the laser sensor can be completed by controlling the movement of X, Y and Z axes of the CMM and several measuring points on the spherical surface can be collected by the sensor, in which the outputs of the grating rulers and the sensor were all recorded. Based on the above and the constraint equation of the spherical surface, a set of over-determined nonlinear equations was set up and the matrix least square method was used to solve the unit direction vector of the laser beam. Finally, a sphere with known diameter was measured by the optical coordinate measuring system at 10 different directions. Through calculation, the measuring errors are all smaller than 0.05 mm, which manifests the effectiveness of the calibration method proposed in the paper and lays the foundation of measuring spatial free surfaces rapidly and accurately.