Shot segmentation is a vital technology that must be resolved firstly in video retrieval and browse. Then key frame extraction will be carried out after shot segmentation to describe main content of shot. An improved algorithm for shot segmentation and key frame extraction based on clustering is proposed, a referenced variable is used in unsupervised clustering to resolve the frame sequence's incontinuity or false segmentation problems which can be caused probably by unsupervised clustering. During key frame extraction, the concept of image entropy is used after shot being segmented into sub-shots. Experimental results demonstrate the efficiency of the improved algorithm in shot segmentation and key frame extraction.