This study proposes an optimized ERFNet lane detection algorithm to reduce the imbalance between the speed and accuracy of current lane detection algorithms based on semantic segmentation. First, an efficient core module is designed; introducing operations such as channel separation and channel reorganization, the number of model parameters and calculations are greatly reduced. Then, the down-sampling is adjusted to increase the single-branch down-sampling process, which improves the model parallelism while reducing information loss. Finally, a feature fusion module is introduced at the end of the encoder, and the receptive field is expanded using dilated convolution to extract differently-scaled lane features. We compare the proposed algorithm with four lane detection algorithms based on semantic segmentation on the CULane dataset. Results show that the comprehensive F1-measure of the proposed algorithm is 73.9% when the intersection-over-union is 0.5, and the inference time per image can reach 12.2 ms, which is superior to the other four models and achieves a good balance between speed and accuracy.