To address the challenges faced in the real-time detection of small-size rockfalls in open-pit mines during the transportation of ores using unmanned carts owing to suboptimal road conditions, intense lighting, and heavy dust, this study proposes a method for detecting small-size rockfalls in open-pit mines based on solid-state lidar. The proposed method employed a double-echo lidar for data acquisition, effectively reducing dust interference and extracting the driving area in front of the vehicle. Subsequently, a ground segmentation algorithm (straight-line fitting) based on fan surfaces was employed to segment the rough and unstructured terrains having slopes. Moreover, a hierarchical grid tree model known as octree was introduced to enhance the efficiency of neighborhood search. Furthermore, the two-color nearest pair method was applied to construct a graph, rapidly generating the clusters. Finally, the concept of adaptive clustering radius ε was adopted for clustering and obtaining the box models of small-size rockfalls. The experimental results demonstrate that the proposed method outperforms the k-d tree-accelerated DBSCAN algorithm, increasing the positive detection rate by 9.61 percentage points and reducing the detection time by 379.77 ms.