To carry out the precise measurement of large-scale complex workpieces, accurately calibration of the stereo vision coordinate system has becoming more and more important. This paper presents a flexible and valid method for calibrating stereo vision coordinate system, which overcomes the complexity and difficulty of the traditional calibration methods. By moving the optical reference bar over the measurement volume, there are three horizontal infrared LEDs and three vertical infrared LEDs with known length which are as feature points, the two cameras capture the images of the feature points. Then, the relative orientation and position of two cameras can be quickly determined with epipolar constraint of the stereo camera through linear algorithm and Levenberg-Marquardt (LM) iteration method. The scale factor of the translation between the two cameras can be calculated by the distance between the two extremity feature points of the optical reference bar. By automatically controlling the light intensity and optimizing the exposure time, the feature points can get uniform intensity and obtain higher signal-to-noise ratio at different position, so the calibration accuracy is improved. The experimental results prove that the proposed calibration method is flexible, valid, and can obtain high calibration accuracy. The maximum error of the large-scale three-dimensional measurement system is 0.18 mm.