Segmentation of human instances is a fundamental problem in human-centered scene understanding and recognition. However, due to the diversity of human body shapes and interactions, spatial relations become complex, posing significant challenges for segmentation tasks. At the moment, most of the mainstream instance segmentation methods rely heavily on the boundary box detection of objects, and thus, are usually unable to effectively separate two highly overlapping objects. In this paper, human skeleton features with complete data annotation are used to provide a priori knowledge for the human instance segmentation task, and a two-stream network structure is proposed to extract skeleton and context features, respectively. The feature fusion module (FFB) then adaptively combines the features from different streams and sends them into the segmentation module, where the final segmentation result is obtained. The proposed algorithm’s average accuracy on the COCOPersons and OCHuman datasets is 59.5% and 56.7%, respectively, which is improved better than other algorithms.