In industrial aluminum defect detection, sparse defect samples always lead to the training overfit and poor generalization. This study describes a defect detection model based on multi-task deep learning. Based on Faster RCNN, a multi-task deep network model is designed, including the aluminum area segmentation, defect multi-label classification, and defect target detection. Then the multi-task loss layer is designed, and the weights are balanced by using adaptive weights to solve the problem of uneven convergence in multi-task training. Experiment results show that with the support of a limited dataset, the proposed method can improve the accuracy of multi-label classification and defect target detection while maintaining the optimal mean intersection over union (MIoU) index of the segmentation task, compared to single-task learning. The method solves the problem of low detection accuracy caused by fewer samples of aluminum defect detection. For multi-tasking application scenarios, the model can simultaneously complete three tasks, while reducing the inference time and improving the detection efficiency.