To solve the problem that motion information in the video is combined to the detection process of the shot boundary, which is to prevent the impact of target movement, camera movement for detection, a shot boundary detection method based on optical flow is proposed. Firstly, motion information in video using optical flow method was detected, which is used to calculate the entropy of adjacent image correction, and a method of optical flow quantization is proposed. The quantization value and corrected entropy is used to determine the continuity of the image, and candidate boundaries are reserved for the presence of a lens switch. Secondly, the image of the photometric information and mutual information between the images are extracted using the difference between the lens structure, and a model matching method is proposed to identify the type of shot. Significant areas of the image are extracted for computing mutual information, which increases the difference between the images and ensures the accuracy of the model matching. The results show that the algorithm can accurately detect the shot boundary, and effectively eliminate the interference of motion.