Cloud detection is the prerequisite of remote sensing image processing and application. It is a widely challenge that the accuracy of cloud detection from remote sensing image is easily influenced by thin clouds and cloud-like ground targets. Therefore, a hierarchical support vector machine (H-SVM) cloud detection algorithm combining grayscale, texture, and frequency features of remote sensing image is proposed in this work. Firstly, a simple linear iterative clustering algorithm is used to segment the remote sensing image into pixel blocks. Secondly, a H-SVM classifier is designed to perform cloud detection on the segmented pixel blocks, where the first layer of the H-SVM preliminarily divides the pixel blocks into “cloud” and “landscape categories”, and the second layer containing two classifiers further classifies the classification results of the first layer and then merges the classified results into three categories of “thick cloud”, “thincloud”, and “land features”. Finally, the classification results are processed using expansion algorithm to get the final cloud detection results. RGB band remote sensing images of GF-1 WFV are selected for verification experiments. It is shown that the method proposed in this study has an average accuracy of 95.4% for the cloud detection in the experimental images, which indicates that the method can be used for cloud detection of remote sensing images in multiple scenarios, and serve the productionand application of remote sensing products.