With the development of digital pathology and artificial intelligence technology, research on the automatic detection of abnormal cervical cells has made great progress. Of the different technologies, object detection technology based on deep learning can simultaneously locate and classify an object, making it a promising application in the field of abnormal cervical cell detection. However, the detection accuracy of abnormal cervical cells still has room for improvement because of the subtle features of abnormal cells that are difficult to extract, small targets that are easily missed, and inaccurate boundary regression. Therefore, in this study, a full-scale feature fusion RetinaNet algorithm combined with attention (AFF-RetinaNet) is proposed for abnormal cervical cell detection to improve the detection accuracy of abnormal cervical cells. In addition, for practical application scenarios, a whole slide image (WSI) inference process based on the AFF-RetinaNet algorithm was implemented for detecting abnormal cells in WSI, which can help pathologists quickly locate abnormal cervical cells in high-resolution WSI and reduce the burden of having to read cervical cytology images.

To improve the accuracy of abnormal cervical cell detection, AFF-RetinaNet is proposed. First, ResNeSt-50 was used as the feature extraction network to extract the fine features of abnormal cervical cells. The structure of balanced feature pyramid (BFP) was then applied to integrate all feature layers at full scale and obtain the global information of the image, which can enhance the semantic information of small targets. Finally, CIoU loss was used as the loss function of the regression branch to improve the accuracy of abnormal cell boundary regression. In the WSI inference process based on AFF-RetinaNet, the WSI was first divided into several patches, and AFF-RetinaNet was then used to obtain the detection results of each patch. Finally, all detection results of patches were integrated, and the non-maximum suppression (NMS) algorithm was used as a post-process to obtain the final WSI detection results.

Figure 1 shows the proposed AFF-RetinaNet for abnormal cervical cell detection. The ResNeSt-50 network was used as the feature extraction network (Fig. 2). Results show that the overall mean average precision (mAP) of the AFF-ResNest-50 increases by 0.8 percentage points compared with the RetinaNet, whereas the mAP of small objects (mAP-s) increases by 1.4 percentage points (Table 3), indicating that the ResNeSt-50 network can improve the ability to identify the features of abnormal cells. BFP was added to integrate all feature layers at full scale and introduce the global information of the image. However, due to the defects of SmoothL1 Loss, the mAP and mAP-s increase by only 0.3 and 0.9 percentage points, respectively (Table 3). Finally, the CIoU loss was used as the loss function of the regression branch to obtain a more accurate boundary regression. Accordingly, the mAP and mAP-s increase by 2.1 and 8.5 percentage points, respectively (Table 3). In a comparison experiment with other mainstream target detection algorithms, AFF-RetinaNet exhibits optimal detection performance (Table 4), further verifying the superiority of AFF-RetinaNet. The WSI inference process based on the AFF-RetinaNet algorithm (Fig. 6) can automatically detect abnormal cells in a high-resolution WSI (Fig. 10) with an mAP of 70.8% in the regions of interest (Table 5).

In this study, a detection algorithm called AFF-RetinaNet was proposed for detecting abnormal cervical cells. ResNeSt-50 was used as the feature extraction network to improve the ability to identify the features of abnormal cells. BFP realizes full-scale feature fusion and introduces the global information of an image to retain more semantic information about small objects. CIoU loss considers additional factors, such as the central distance and shape relationship, which improve the accuracy of abnormal cell boundary regression. In addition, the WSI inference process based on the AFF-RetinaNet can automatically detect abnormal cells in a high-resolution WSI. Experimental results show that the mAP of AFF-RetinaNet on the abnormal cervical cell dataset is 83.4%, and the mAP-s reaches 24.4%, which are 3.2 and 10.8 percentage points higher, respectively, than those of the baseline RetinaNet and better than those of the other mainstream object detection algorithms. The detection results of the cervical WSI inference process based on AFF-RetinaNet reveal an mAP of 70.8% in the regions of interest. Our study shows that AFF-RetinaNet can improve the detection accuracy of abnormal cervical cells and that the WSI inference process based on AFF-RetinaNet can assist pathologists in cervical cancer screening, giving the proposed algorithm significant clinical application value.