A key frame extraction algorithm of sign language based on compressed sensing and speed up robust features(SURF) feature is proposed to recognize the real-time, large vocabulary sets and continuous sign language videos efficiently and accurately. The sign language videos are reduced to the image features of low dimensional and multi-scale frame with compressed sensing. The segmentation of sub lens is completed by a adaptive threshold value, and a large number of sign language frame data are processed. We use SURF feature points to complete the feature matching, and the SURF frame similarity curve is drawn for extracting the key frames. In the pre-processing stage, we use the HSV space adaptive color detection to abstract the sign language area. Experimental results show that the key frames extracted by the proposed algorithm have high accuracy, and the proposed algorithm has the ability to process large amounts of complex data.