With advancements in science and technology, welding technology has progressed from manual to automated and intelligent welding. The widely used weld tracking technology based on laser vision can improve the ability of welding robots to perceive their environments, with the added advantages of non-contact and high precision. However, in real-time weld tracking, the collected weld images are often severely affected by strongly reflected light, splash, and arc noise. Therefore, laser stripes are accurately and quickly extracted from images containing a large amount of noise, and then obtaining weld information from them is a prerequisite for high-quality welding. To improve the weld location accuracy and the image processing speed in the weld tracking process, this paper proposes a lightweight multi-task deep learning model that combines laser strip segmentation and weld feature point location. The model consists of an encoder and a decoder. The laser fringe segmentation subtask and the weld feature point location subtask share the encoder backbone network. The decoder includes a laser fringe segmentation branch and a weld feature point location branch based on differentiable space-to-numerical transformation (DSNT). The entire model is designed in a lightweight manner, and it simultaneously adopts relevant information between multiple subtasks to further improve the performance of each subtask. In summary, we expect that the designed deep learning model can achieve accurate and rapid acquisition of weld features during the welding process.

In order to improve the weld location accuracy and image processing speed in the weld tracking process, a lightweight multi-task deep learning model combining laser strip segmentation and weld feature point location is proposed. The proposed model adopts the parameter hard sharing mechanism in multi-task learning such that the model uses fewer parameters. Specifically, the model consists of an encoder and a decoder. The encoder completes the feature extraction of weld position and edge information, while the decoder implements the output of the laser stripe segmentation and feature point location subtasks. The encoder network adopts the concept of a more efficient bilateral segmentation network, including context and spatial paths. The context path realizes the extraction of high-level semantic features of the image, and the spatial path provides edge detail information. In addition, to make up for the loss of detailed information, the spatial path is supervised with detailed information. To utilize the information that the weld feature point is located on the laser fringe, multi-stage supervision is adopted to make the encoder structure learn the characteristics of laser fringes. Therefore, the structure of the laser fringe segmentation subtask in the decoder only contains a simple convolutional layer and an upper sampling layer, which can realize the output of laser fringes. The DSNT module is introduced into the feature point location subtask to realize the fusion of the Gaussian thermal diagram method and the fully connected layer method so that the model is completely differentiated and has the spatial generalization ability of the Gaussian heat map method.

The results of laser fringe segmentation on images disturbed by noise demonstrate that our model exhibits good segmentation accuracy, and the detail information supervision of low layers can further improve the segmentation accuracy (Fig. 8); in addition, our model achieves a good balance between accuracy and speed (Table 2). The location results of the weld feature points show that DSNT can accurately locate the feature points of the weld under different noise interference conditions (Fig. 9). Through an experiment where the output layer structure of the network was changed, we verified that compared with the Gaussian thermal diagram method and fully connected layer regression method, the DSNT method can achieve subpixel-level location with minimal errors (Fig. 10). By changing the output structure of the decoder, it is experimentally verified that the laser stripe segmentation subtask can improve the performance of the weld feature point location subtask (Fig. 11). Finally, the experimental results verified that, compared with various deep learning models, the proposed network model can complete the segmentation of laser stripes and the localization of feature points while maintaining the inference time (Table 3).

In this study, a multi-task learning model for laser fringe segmentation and weld feature point location is proposed for weld images with multiple noises. Using detailed contour information to supervise the characteristics of the lower layer can improve the segmentation performance of laser fringes. By changing the network layer of the feature point location part, the DSNT module exhibits a higher weld feature point location accuracy than the Gaussian heat map method and the fully connected layer regression method. The multi-task learning method improves the accuracy of the location of weld feature points. In addition, the inference time of the proposed network can meet the real-time requirements of image processing in weld tracking. In summary, our model can effectively handle all types of welding noise and complete the feature extraction of welds, demonstrating good application prospects in automated welding.