Aiming at the problem of low accuracy in autonomous driving systems with limited computing resources and multi-task driving perception algorithms, a multi-task driving perception algorithm with improved HybridNets is proposed. EfficientNetV2-S is selected as the backbone network of this algorithm to reduce the number of parameters, improve training speed and recognition accuracy; combine depth-separable convolution and use shuffle-channel convolution to reduce the amount of model calculation; use three independent decoders to solve problems of different difficulties, and add the A2-Nets dual attention machine block between the backbone network and the Neck end to fully extract global features. Compared with the basic network HybridNets, the mAP50 of this model can reach 79.8% in the vehicle detection task, an increase of 2.6%; the mIoU in the drivable area segmentation task can reach 91.8%, an increase of 1.2%; and the IoU in the lane line detection task can reach 32.55%, an increase of 0.93%. The running speed reaches 38 FPS. Experimental results show that compared with existing methods, the accuracy of the proposed method is greatly improved.