Segmentation of surveillance video into motion segments is the basis and premise of video synopsis. The existing video segment segmentation methods are complex in their implementation and computationally intensive, which has a detrimental effect on the real-time performance of video synopsis. To address these issues, a motion segment segmentation method of surveillance video based on the spatio-temporal flow model is proposed. The proposed technique only sparsely samples the boundary pixels of the video surveillance area to create the video spatio-temporal profile. On this basis, background modeling is used to extract the targets from the spatio-temporal profile. A spatio-temporal flow model for moving objects entering and exiting the visual surveillance area is subsequently constructed. Finally, the model is modified according to the moving target's feature matching, and the accumulative spatio-temporal flow curve of the video is obtained, from which the motion segments are then segmented. Experimental results show that the presented method not only ensures video segmentation accuracy but also dramatically increases the speed of video motion segment segmentation.