As rail images show uneven gray distribution, general image segmenting methods can not accurately segment rail images. To address this issue, this paper presents an improved Otsu method using weighted object variance(WOV) for rail image segmentation to separate the defect from its background. Firstly, the property of a rail image was analyzed and the problems of the Otsu method and other global threshold methods for segmenting rail images were summarized. Then, the Otsu method was improved. By taking the cumulative probability of defect occurrence for the weighting, the object variance of between-class variance was weighted, and the threshold will always be a value that locates at two peaks or at the left bottom rim of a single peak histogram. Finally, the misclassification error (MCE), the detection rate and false alarm rate of the defect image were calculated to validate the effectiveness of proposed method. The experimental results demonstrate that the improved Otsu method accurately segments various kinds of rail images and the MCE value is close to 0. As comparing to the Otsu method, other improved Otsu method and maximum entropy threshold method, the proposed method provides better segmentation results, the detection rate and false alarm rate for the rail defected image are 93% and 6.4% respectively. It is suitable for the applications in machine vision defect detection in real time.