The traditional segmentation algorithms for straw coverage detection basing on thresholds or texture features were difficult to get rid of the disadvantages of low accuracy, high complexity and time-consuming, and the effect of segmentation on complex farmland scenes containing a lot of interference factors was not good. Therefore, this paper proposed a semantic segmentation algorithm (DSRA-UNet) with high accuracy, a small mount of training parameters and high running speed. Combined with UNet′s symmetric codec architecture, this algorithm used standard convolution in shallow feature maps, and depthwise separable convolution in deep ones. Residual structure was built in each layer to increase the network depth,which can reduce the number of parameters and improve the accuracy at the same time. In addition, the global maximum pooling attention mechanism was added during the skip connection process to further improve the segmentation accuracy of the network. The algorithm was verified on the straw datasets, and the experiment results showed that the mean of intersection over union reached to 94.3% in the proposed algorithm of this paper. The number of training parameters of the algorithm was only 0.76 M, and the test time of single picture was less than 0.05 s. The algorithm could accurately segment the straw and soil, and separate the interference information in the complex environment, especially solving the shadow problem in image.