There are a variety of functional proteins localized on the cell membrane that participate in many crucial cellular processes, such as signal transduction and transmembrane transport. The spatiotemporal distribution of specific membrane proteins largely determines their activity states and functions. It is known that the sizes of membrane proteins and the distances between them are both on a nanometer scale. Owing to diffraction limits, traditional optical microscopy cannot provide the spatial distribution of membrane proteins at the single-molecule level. Therefore, imaging techniques with strong specificity and high resolution are urgently required to reveal the precise spatial distribution of membrane proteins. Nowadays, single-molecule localization microscopy (SMLM) offers new opportunities to resolve the detailed distribution information of membrane proteins at the nanoscale, while the great improvement in spatial resolution also presents higher demands for accurate clustering segmentation of images. Density-based spatial clustering of applications with noise clustering (DBSCAN) is one of the most commonly used clustering methods; however, it shows relatively poor performance in clustering segmentation in SMLM images of membrane proteins with heterogeneous density. To address this issue, we propose a novel clustering method using a combination of a multi-step DBSCAN and a hierarchical clustering algorithm. This improved clustering method is based on the traditional DBSCAN method, which combines area threshold analysis and hierarchical clustering.

In the present work, we improved the traditional DBSCAN method by introducing a variable neighborhood radius and hierarchical clustering to perform precise image clustering segmentation in the original image (Fig. 2). First, we inputted a relatively large parameter (ε₁, M₁) to perform the DBSCAN calculation. Owing to this relatively large parameter, the excessively discrete points in the original image were removed as noise points. Meanwhile, some of the close-point clusters merged together. Subsequently, the area of each preliminarily identified cluster was calculated and divided by the average area for normalization. Based on the acquired normalized values, we selected an appropriate threshold parameter for extracting clusters with a relatively large area. Subsequently, secondary DBSCAN was performed by the input of a smaller or equal parameter (ε₂, M₁; ε₂≤ε₁). For each point cluster extracted in the second step, the calculation was looped from ε₂ to ε₁. The parameter showing the maximum number of divisible point clusters in the output during the looped process from ε₂ to ε₁ was selected as the clustering parameter for the next hierarchical clustering. Finally, we combined the above two DBSCAN results to obtain the final clustering segmentation result.

We tested this improved clustering method on both simulated and experimental SMLM data. For the simulation datasets, we chose the D31 and S2 datasets from previous studies as our test objects (Fig. 4). The purity of the improved method on the D31 dataset was 95.64% (86.52% for the traditional DBSCAN method), and the adjusted Rand index was 0.9186 (0.6463 for the traditional DBSCAN method). In addition, the silhouette coefficient and noise ratio were used to analyze the two datasets. Compared with the traditional DBSCAN method, the silhouette coefficient of the improved method significantly increased, and the noise ratio decreased (Table 1). For the S2 dataset, the improved method also exhibited a more accurate segmentation effect than the traditional DBSCAN method. The identification purity of the improved method for the S2 dataset was 95.52% (77.38% for the traditional DBSCAN method), and the adjusted Rand index was 0.9128 (0.6777 for the traditional DBSCAN method). The silhouette coefficient and noise ratio increased and decreased, respectively (Table 1). For the experimental SMLM data, we tested the clustering segmentation effect of the improved method on the uniform, random, and non-uniform SMLM images of membrane proteins (Fig. 5). Similarly, the improved clustering method has a higher accuracy and silhouette coefficient, and a lower noise ratio (Table 1). However, it is regrettable that the time consumption of the improved clustering method is higher than that of the traditional DBSCAN method for both the simulated and experimental datasets (Table 1).

Based on the characteristics of the point clusters in SMLM images of membrane proteins, we proposed a novel clustering method that combines area threshold segmentation and multi-step clustering segmentation based on the traditional DBSCAN algorithm. When we applied this method for the image segmentation of simulated datasets as well as experimental SMLM data of membrane proteins, the obtained parameters, including purity, adjusted Rand index, silhouette coefficients, and noise ratio, were generally improved compared with those of the traditional DBSCAN method. On the premise of accurate clustering recognition of super-resolution images and a certain noise reduction ability, the localization information of each cluster can be preserved as much as possible. Our method exhibites a good clustering segmentation ability, especially for SMLM images of membrane proteins with heterogeneous densities. This improved clustering method provides novel insights into the segmentation of membrane protein SMLM images, which is expected to facilitate research into the nanoscale spatial distribution of various membrane proteins.