Objective It is essential to initially establish a precise recognition model to achieve accurate control for a penetration state in laser welding. Although the recognition method of the penetration state using visual signals is widely proposed, there are still interferences, such as plasma, vapor, and spattering, in the laser welding process. Besides, there is no significant change in adjacent penetration state. These are the problems in vision-based recognition of the laser welding penetration state. Feature engineering and deep learning seem to be the only methods to solve these problems. Although the feature engineering-based method is interpretable, it requires many subtasks to decrease overall recognition efficiency. Also, the value of data cannot be fully developed. On the other hand, the deep learning-based method realizes an end-to-end recognition from the original image to the penetration state. It improves the overall recognition efficiency and data value. The deep learning-based method becomes a major research focus because of integrating intelligent technologies into manufacturing systems. However, deep learning-based methods require a large amount of data, because fewer data result in overfitting. The boundary between adjacent penetration states is unclear, making it difficult for supervised learning methods to be applied. Inspired by the fact that skilled welders consider asymptotic information in deciding on the welding process, we propose a laser welding penetration state identification method that incorporates timing information. The timing information is expected to improve the determining factor of a deep learning method for the weld penetration state and increase the amount of data.

Methods The frame of the proposed method consists of a spatial feature extraction module (SFEM) using a convolutional neural network (CNN), a time domain feature extraction module (TDFEM) using a bi-directional long short-term memory (BiLSTM) neural network, and a classification module (CM) using a SOFTMAX function. In the SFEM, we used two convolutional layers and two max-pooling layers to extract the input image sequence's spatial features. Afterward, we applied TDFEM to extract the features in the time domain from the input sequence. In the TDFEM, the feature sequence was simultaneously input into the forward LSTM and reversed the LSTM to obtain the forward and reverse outputs. Then, we summed the forward and reverse outputs of the same input as the final output of the current input. In the CM, we first input the spatiotemporal features into the fully connected network for dimensionality reduction. Subsequently, we mapped the low-dimension features to categorical probabilities using the SOFTMAX function. For data acquisition, the optimum penetration condition was obtained through a welding test. We incremented the laser power corresponding to the optimum penetration state, and then decremented it to obtain excessive penetration and incomplete penetration conditions.

Results and Discussions Figure 5 shows that the CNN-BiLSTM method's accuracy converges to 1 after 5000 iterations, whereas the CNN method's accuracy only converges to approximately 0.93. Using the CNN-BiLSTM method, the difference in accuracy on training and validation sets is insignificant, suggesting no overfitting using the proposed method (Fig. 6). The identification accuracy and overall evaluation index of the CNN-BiLSTM method reach 99.26% on the test set, much higher than the those of the conventional CNN method. Although various CNN-LSTM method indicators are about 97%, the CNN-LSTM method only considers the previous information of current input in the time domain without considering subsequent information of current input in the time domain. The proposed method takes only 9.43 ms to identify a single image in a PC (Table 2). The CNN and CNN-LSTM methods misclassify the penetration state as the true label. Moreover, Fig. 7 shows that the proposed method can suppress misclassification. In this paper, the training process's convergence tends to be consistent when the learning rate (LR) is close to 10 ^-3. The model does not converge when the LR is 10 ^-5, suggesting no overfitting. When the optimizer (OM) is set to Adagrad or Adam, the training process's convergence is similar to that of the stochastic gradient descent method applicable to this paper. The proposed method could not converge within three epochs when the Abadelta OM is used because the Adadelta easily falls into the local optimum in the middle and later training stages (Fig. 8). Table 3 shows that all accuracy metrics of CNN-BiLSTM are above 97.66% when the LR is around 10 ^-3 or OM is replaced, which suggests that the proposed method is robust.

Conclusions In this paper, timing information was not considered in deep learning-based methods for penetration state recognition. Our proposed method, CNN-BiLSTM, can adaptively extract spatiotemporal context information, as the method demonstrated good convergences and stability. The introduction of temporal information can indirectly play the role in data augmentation, and the proposed method does not overfit. The overall recognition accuracy of the proposed method on the test set is 99.26%. As such, the proposed method meets the standard requirements of vision-based laser welding in terms of penetration condition monitoring. Furthermore, the method is robust to changes in LR and optimizer. Although the proposed method has many advantages, the following aspects still need to be focused in future research. Better classification of penetration states will be a future interest choice in terms of research objects. Also, making the network structure fit the parallel computing system will be a good future research direction. In terms of model optimization, making the timing model lightweight without reducing the accuracy will be another good option.