Hyperspectral image clustering has always been a hot topic in the field of image processing. Spectral clustering algorithm, as one of the most popular clustering algorithms, is widely used in hyperspectral image clustering. However, due to the large computational complexity of the spectral clustering algorithm, it is difficult to process large-scale hyperspectral image data. Because the binary tree can select anchor points very fast, the spectral and spatial characteristics of a hyperspectral image are fully utilized to ensure the clustering performance and reduce the computational complexity based on the binary tree anchor graph. However, the clustering algorithm generally adopts the kernel clustering method, therefore it is inevitable to introduce parameter adjustment. Thus, based on the selection of anchor points in the binary tree, we proposes a hyperspectral fast clustering algorithm based on the binary tree anchor graph. This algorithm innovatively applies the method of binary tree anchor selection and coreless clustering to the hyperspectral images. First, the binary tree is used to select some representative anchor points from the hyperspectral data. Second, a coreless similarity map is constructed based on these anchor points, which effectively avoids the artificial adjustment of the thermonuclear parameters to construct the similarity map. Third, the spectral clustering analysis is performed to obtain the clustering results. Finally, this algorithm is used for hyperspectral image clustering. This algorithm not only improves the clustering speed, but also reduces the necessity of original thermonuclear parameter adjustment. The experimental results show that the proposed algorithm can obtain better clustering accuracy in a shorter time compared with the traditional clustering algorithm.