In order to improve the segmentation accuracy of single stage instance segmentation and improve the situation of missed and wrong detection of small targets, an improved YOLACTR algorithm is proposed based on YOLACT algorithm. The algorithm first used the combination of CNN and Transformer to design a new head prediction network to further extract features, and used two-way attention to correlate the mask information of the same instance and distinguish the mask features between different instances. It paid attention to the correlation information around the feature points, making the prediction of the detection box more accurate. Then the mask branch was formed by the combination of multi-level up sampling module and the designed CS attention module, which integrated a variety of different scale information. Then the CS attention module was used to pay attention to different scale information. On the MS COCO data, compared with YOLACT algorithm, YOLACTR algorithm improves the detection accuracy of box and mask by 7.4% and 2.9% respectively, and improves the detection accuracy of small targets by 18.9% and 13.5% respectively. Experiments results show that YOLACTR algorithm can improve the accuracy of detection, segmentation and classification in multi-target complex scenes, which improves the problem of missed and wrong detection of small targets and overlapping targets.