The qualities of electronic materials can have a considerable impact on the performance of electronic devices^[1-3]. Even a nanoscale defect could result in the failure of a device^{[4, 5]}. Therefore, comprehensive analysis of the electric material is indispensable. The transmission electron microscope (TEM) has been widely applied for the structure-property study of materials and devices due to its high-spatial resolution and versatile characterization abilities^{[6, 7]}. With the developed aberration correction technology, the spatial resolution of the TEM has been improved to 0.5 Å. This allows scientists to directly observe the atomic configuration of materials^{[8, 9]}. In addition to morphology, TEM can also provide structure, composition, and even valence state^[10-13]. The recently developed in situ TEM technology also provides a real-time method to characterize and manipulate materials with external stimuli, such as electrical, mechanical, thermal, optical fields, and liquid/gas environments^{[6, 14-17]}.

The sample information obtained by TEM characterization is presented in the form of images^[18-22]. Great progress has been made in the acquisition and processing of TEM images, as shown in Fig. 1. In the early stage, the TEM images were captured by camera films that have to be developed to get the sample information. However, this process is complicated, time-consuming, and requires training to obtain a qualified image^[23-27]. The data is not digitalized and the image analysis has to be calculated by hand, which results in low accuracy and efficiency. With the improved performance of photosensitive devices and the increased pixel density, CCD (charge-coupled device) cameras replaced film cameras for TEM image recording. The CCD camera converts electronic signals into photon signals through scintillators. Then, the photon signals activate the silicon epitaxial layer and generate charges. The accumulated charges are converted into voltage and stored in memory, forming digital images. The digitalized TEM image can easily be processed by professional software, such as digital micrograph and so on, which greatly improves the accuracy and efficiency of TEM data analysis^[28]. The forthcoming TEM camera based on complementary metal oxide semiconductor (CMOS) could directly detect electron signals without a scintillator^[29-32]. This dramatically improves the efficiency and avoids the loss of signals from scintillators, which promotes the quality of imaging, especially in the aspects of signal-to-noise ratio and low dose imaging^{[33, 34]}. TEM methods have also been adopted to study low dimensional materials^[35-39]. In addition, artificial intelligence (e.g., machine learning) will be widely applied to automatically process a large quantity of in situ TEM images with high temporal resolution.

Unlike the color-scaled images in a natural scene, TEM images that are formed by an electron signal are grayscaled maps. In addition, the TEM image has high statistical noise, low contrast, and blurred boundaries, resulting from conditions such as the thickness of the sample, the sensibility of the materials to electron beam, and so on. To mitigate the damage of the sample induced from the electron beam irradiation, a common solution is to reduce the shooting time at the sacrifice of resolution^[40].

Analysis of TEM data is time-consuming and requires specialized knowledge, which mainly results from two aspects. First, the features in TEM images are usually complex and vague, which increases the difficulties for fast and accurate information extraction. Second, a large number of TEM images can easily be obtained for each TEM experiment, especially for video streams that are recorded during in situ TEM experiments. Even at the present stage, more than a thousand images can be acquired within several minutes. These issues have led to the demand for a way to efficiently analyze large and complicated TEM datasets^[41-43].

The resurgence of machine learning has greatly impacted image recognition and offers an excellent opportunity for automated TEM data analysis^[44-49]. Machine-learning methods are mathematical models that are used to help a computer learn to automatically process data, imitating the methods that humans use. Machine learning is generally divided into three subcategories: supervised, unsupervised and reinforcement machine learning. The difference between supervised and unsupervised methods lies in whether or not the dataset is labeled. The reinforcement method learns from trial and error through a feedback loop. Neural networks are a fast-developing machine-learning method that provide high accuracy in image recognition and require less human intervention. The first convolution neural network (CNN) model, LeNet, was reported by LeCun et al. in 1989, and adopted local response and pooling to enhance performance of the network^[50]. Subject to the computer power at that time, machine learning did not become popular until the AlexNet came out in 2012. The network imported dropout and data augmentation to avoid overfitting, and rectified linear units (ReLUs) were used as a new activation function, which was a big step forward in computer vision^[51]. Recently, machine learning based methods have been successfully used in medical and natural image recognition^[52-54].

In the field of TEM high-throughput images analysis, tasks concern segmentation or classification of specific characteristics of interest from material images. Neural networks for automated analysis of TEM images mainly include a CNN algorithm, a fully convolution network (FCN) algorithm and a U-Net algorithm. The CNN is determined by three components, the convolutional layer, pooling layer and a fully-connected (FC) layer. The first two layers produce feature maps and reduce dimensionalities. The last, FC, layer produces a one-dimensional vector for classification tasks. Unlike CNN scarifying of the spatial information of input images, the FCN performs classification for each pixel of the image by replacing the FC layer with the convolutional layer. The U-Net is an improved FCN structure with encoder and decoder networks, which has symmetric skip-connected contracting and expansive paths to capture features and for accurate localization. The U-Net can be trained by the available labeled samples with higher efficiency. U-net has been widely adapted in medical image identification, due to its efficiency and precision. U-net is comparatively more adaptable for TEM image segmentation tasks, which is similar to that of medical images tasks.

Efforts have been made to integrate automatic approaches with imaging processing to cope with the obstacles existing in TEM image analysis^[55-61]. As shown in Fig. 2, the publication trend suggests a sharp increase number of studies using the machine-learning method for TEM image analysis. Along with the development of computing power, the automatic analysis of TEM imaging by machine learning is very promising and still in its initial stages, and deserves more effort. Machine learning has contributed to the analysis of the nanomaterial images by TEM characterization in the aspects of defect, morphology, structure, and spectra, as shown in Fig. 3^[62-67]. The main information of the typical works that have been published recently is summarized in Table 1.

In this review, the recent developments of the TEM data analysis based on machine learning are summarized. We first present the background and technique evolution of the TEM data acquisition and analysis. Then, TEM data analysis is discussed in detail from four aspects: morphology, defect, structure, and spectra by the application of various neural networks. Finally, in the conclusion we state the challenges of artificial intelligence assisted TEM data analysis.

Meanwhile, 2D materials with ultrathin geometry, large specific area, and rich physical properties are widely applied in the fields of electronics, sensors, energy storage and conversion^[73-79]. The study of local defects has for a long time been an important research field in material science because it can greatly affect the physical and chemical behaviors of 2D materials, even with a limited number of dopants or vacancies^[80]. The defects of a 2D material could influence the semiconductor carrier type, the catalytic, and optical properties^[81-85]. In terms of defect analysis, determining atom position is indispensable^[86]. Scanning transmission electron microscopy (STEM) is an appropriate characterization method for atomic-defect investigation due to two aspects: the sub-angstrom spatial resolution and the contrast sensitivity to the square of the atomic number^{[87, 88]}. STEM characterization uses a focused electron beam to probe each atom of the TEM sample and then collect the scattered electron signals to form the image contrast. An element with a higher atomic number can diffract the incident electron beam more effectively, which results in higher contrast. For the traditional manual processing method, processing a large number of in situ STEM images is nearly impossible. Machine learning based approaches are highly effective and are increasingly applied in the tasks of atomic defects localization, identification, and classification.

Ziatdinov et al., for instance, take advantage of a weakly-supervised FCN to identify defects, including vacancies and doping atoms in STEM images. The FCN model contains an encoder-decoder structure, which turns the last fully-connected layer of the conventional CNN into a convolution layer, and outputs the probability mapping of each pixel corresponding to various atoms or defects; as shown in Fig. 4(a). This approach could start with a few labeled general defects and can identify complex defects, even those not in the training set, requiring less human intervention^[72].

The quantity and quality of the labeled TEM dataset are critical for training a network with high accuracy. However, there is still lack of high-quality labeled TEM datasets. As an alternative to labeled experimental data, the simulated TEM data with accurate ground truth is easier to obtain. The multislice calculation-based algorithm is usually applied for TEM data simulation, including STEM images^{[60, 64]}. The simulated STEM images are generated to train the model, and experimental images are used as the test dataset. After the center and contour of individual atoms, the atom positions and the chemical bond configurations can be explored; as presented in Figs. 4(b) and 4(c). By analyzing each frame of the in situ STEM images, the defect evolution with the variation of time is quantitively studied Figs. 4(d)−4(f).

Using the FCN algorithm, the atom positions and types in a lattice are recognized with high accuracy, enabling effective defect determination. The presented algorithm can be extended to other automatic TEM image analysis that relates to the atomic position.

Similar to medical images, TEM images usually suffer from background noise and vague boundaries. According to the advanced algorithm for medical image recognition, reducing background noise during the image analysis is required.

To reduce the raster distortion during STEM image acquisition, fast imaging speed is required. This results in a low signal-to-noise ratio (SNR)^[64]. Improving the SNR before applying the CNN-based algorithm for STEM image processing could increase the accuracy of the results. Yang et al. innovatively adopt an automated method based on two neural networks to improve the image SNR and complete the image segmentation of atomic defects and dopants in 2D transition metal dichalcogenides (TMDs)^[64]. A CNN-based algorithm (supervised learning) with a dilated convolutional kernel is first applied as a denoiser to the raw experimental images. Simulated ADF STEM images obtained by multislice calculations are used to get a high-quality training dataset, and the noise signal is simulated by the Poisson distribution. Using this method, the SNR and the atomic contrast of the experimental STEM image are enhanced, as shown in Figs. 5(a)–5(c). From high-contrast input images, various defects (including atom doping and atomic vacancies) are clearly identified in the V-WSe₂ by a U-Net algorithm with an accuracy of up to 98%. In addition, the dynamic behaviors of different defects under e-beam stimulus can also be unveiled, as shown in Figs. 5(d) and 5(e).

The U-Net model has been widely used in processing medical images and it shows excellent performance in defect classification. The U-Net algorithm has skip connections and concatenation operation that could identify features of different scales and achieve good accuracy with a small dataset; as shown in Fig. 5(f)^[52]. In addition, U-Net adopts an overlap-tile strategy, which generates no overlapping range and conducts a seamless segmentation to input of any size. This strategy benefits the learning efficiency, especially in scenarios with limited annotated training datasets or unscalable raw images. Except for the point defect recognition, a line defect (e.g., a nanopore border) can also be identified and analyzed by the U-Net algorithm; as shown in Figs. 5(g) and 5(h)^[61]. The CrTe₂ nanopore healing process induced by the synergistic effect of electron beam irradiation and heating field is captured by in situTEM. Over 1300 TEM images that were labeled by hand are used for U-Net training. After 100 epochs training, the DICE metric reaches 93.17%, showing an effective line defect recognition by the U-Net algorithm.

The rapid development of machine learning aided TEM data recognition makes the real-time characterization and analysis of defects during the in situ TEM operation possible. The immediate feedback of the defect evolution information could further guide thein situ TEM experiment along a planned road, which accelerates material mechanism discovery with high efficiency.

Many efforts have been made to study the morphology of nanoparticles (NPs), which govern the physical and chemical properties of the nanomaterial. The morphology of a sample reflects the types of its exposed surfaces, which has an impact on its catalytic activity^{[89, 90]}. The morphology evolution process also unveils the stability or growth kinetics of the NP^[91-97]. TEM can characterize the morphology of nanomaterial with atomic resolution. The 2D projections of 3D particles that are obtained from TEM are usually studied for their morphological features. In general, NP analysis includes three stages: morphology capture, filtering, and shape classification. The foreground and the background need to be separated to locate and isolate the individual particles. Then, in the dataset of detected targets, filtering is required to eliminate the superimposed or abutted particles that may be wrongly recognized and disturb the statistical result. Finally, the particles are put through the classification process and the distribution map is presented. The descriptors, based on which the particles are divided, primarily include solidity, area, convexity, eccentricity, circularity, aspect ratio, and angular distance. Two approaches are introduced as the determined attempts of NPs.

A completely unsupervised method, called AutoDetect-mNP, was introduced by Wanget al. to identify the shape attribution of Au NPs (Fig. 6(a)). To differentiate the individual particles and the background, the algorithm adopts K-means image segmentation. During the TEM imaging process, Au particles (due to their high mass and high lattice packing efficiency) show intense interaction under an e-beam from a TEM, thus generating a strong contrast in the images. Partly because of this high contrast between the foreground and the background, the adopted simple computer vision approach proves sufficient for this stage. The nanorods are classified after going through a filtration for convexity and solidity. For the classifier, this study uses a combination of K-means clustering and naive Bayes classification to serve the purpose of being totally unsupervised; shown in Fig. 6(b). As a result, this approach has a great performance for both efficiency and accuracy when compared to the previous methods for the analysis of Au nanorods. It also reduces the bias by reducing human input. However, when adapted to materials that show lower contrast, such as palladium (Pd) and cadmium selenide (CdSe) particles, the unsupervised K-means image segmentation may not be good enough for identification.

A supervised method is a promising solution to pursue high precision in machine learning based NP detection. Leeet al. explored an approach based on a genetic algorithm (GA) to analyze the morphological properties of NPs, with a precision as high as 99.75%; as shown in Fig. 6(c)^[70]. Using natural selection for reference, the GA filtrates the genes according to a specified standard, and only those with the best performance can be the survivors. The leftover genes go through the same course of evolution, mutation, and filtration. The surviving genes are considered as the optimal result. In this study, locating the NPs is not only the case, but properly separating or deserting the superimposed particles based on the extent to which they are abutted is also needed. Various imaging processing methods (i.e., thresholding and watershed transform) can be used to compose the variables. The aim is to explore the optimal variable-variable combination utilized in the morphology analysis in the NP systems. High-scoring images (genes) of every round are stacked in the algorithm to pile up the detected NPs, and thus augment the statistical significance. After the images are filtered with the GA algorithm and processed with the optimal composition of various imaging analysis techniques, they are clustered and their distribution is analyzed statistically to be ready for further research.

Yao et al. introduced a U-Net model to extract information of physical and chemical properties^[63]. Their machine model was fed with the training set constituted by simulated liquid-phase images. Beer’s law, Poisson noise, and Gaussian noise are considered to simulate the liquid-phase imaging under e-beam. First, the boundaries of NPs in each frame in the TEM video are located, based on which the intensity of four categories of gold nanorods and background is calculated; as shown in Figs. 6(d) and 6(e). Second, during etching, the nanorod boundaries are tracked and marked colored to their curvature. The height and etching rate are monitored in real time. Finally, from the statistical analysis of the big dataset, three stages are concluded, showing how the etching rate changes. Its interrelation with curvature is shown in Figs. 6(f) and 6(g).

In the liquid-phase TEM video, the contrast fluctuation appears to be a primary challenge. The different thickness and orientation of each nanorod result in contrast variation, even in the same image. Even for the same particle, the motions at the time scale lead to a shift in the intensity diagram. Such fluctuation cases happen randomly and inevitably, and have been longstanding obstacles in liquid-phase TEM movie processing. Due to the unique skip-connection structure, the U-Net algorithm includes various factors, such as intensity, morphology, and local surroundings in the process of segmentation, while the previous thresholding approaches include only the intensity of the pixels. U-Net structures are suitable in segmentation tasks for liquid-phase TEM images that have contrast fluctuations and low SNR. The advantages of the application of U-Net models are the robust segmentation, high tolerance over low-quality images, intensity fluctuation and many other defects of the sample. The robust segmentation of the U-Net favors the precise and efficient tracking of boundaries and analysis of the particle properties.

For the morphology of materials, the thickness is closely related to the intrinsic properties, such as bandgap and optical properties^[98-100]. Both HRTEM and the recently developed 4D-STEM images contain the thickness information of the TEM sample. The HRTEM and 4D-STEM are TEM imaging modes with atomic spatial resolution. The HRTEM uses a parallel electron beam to form images, and the phase contrast is generated from the interference of electrons that interacted with and without the sample atoms. The 4D STEM image uses a pixelated signal detector to get the conventional 2D STEM image and 2D diffraction pattern at the same time to form the 4D-STEM data. Compared with the traditional STEM, 4D-STEM provides more information for sample property extraction^[101-103].

Since the sample is projected as 2D in TEM images, the recognition of column heights out of 2D images has been a popular topic. Here, two works are introduced to evaluate the atom numbers in the atom columns that present the thickness of the sample. Two different convolutional networks, a regression CNN (r-CNN) and a classification CNN (c-CNN), are used in these two works for the recognition of HRTEM and STEM images, respectively. The two models are trained and applied to estimate the column heights of metallic NPs. The results of the two methods are compared to clarify their suitability.

First, Ragone et al. detect the atomic column heights of metal particles with two automated methods based on CNN^[69]. Simulated TEM images with the labeled ground truth are calculated by QSTEM^[104]. The brief architecture and the result example of the column height prediction of the model are shown in Figs. 7(a) and 7(b). Based on HRTEM images, both the networks determine the heights based on the values of pixels. In the classification CNN model, the output of the network is a probability map of the thickness of NPs, indicating the discrete classes of the values of their heights. In this study, a maximum height of 15 atoms is considered to be sufficient, and the output classes of thickness are from 1 to 16. The so-called regression-based model is not usually attributed to a standard segmentation task. Unlike the discrete classes in the classification output, the pixel values of TEM images are deemed to have a continuous decrease from the column’s center to background. After being trained with simulated TEM images, the model correlates each pixel to a value of the predicted height. From the comparison of the two methods, the regression-based model is reported to be approximately 10% more accurate than the classification model. The contrast of HRTEM is sensitive to the degree of defocus. The influence of defocus to image recognition accuracy is evaluated as shown in Fig. 7(c). The low defocus and low dose images have relatively low accuracy.

Second, in Zhang et al. the convergent beam electron diffraction (CBED), namely 4D STEM, is adopted to assist the CNN-based thickness detection^[60]. As is shown in Fig. 7(d), the positive-averaged CBED (PACBED) pattern of Sr particles is obtained and used to generate a thickness prediction map through a CNN. Both the c-CNN and the r-CNN determine the number of unit cells overlapping on one column. In Fig. 7(e), the result of c-CNN is listed in the red frame and the r-CNN is in the blue frame. A comparison of the two models shows that both models are sufficient on sample thicknesses within 40 nm, and have varying degrees of falling in accuracy as the thickness increases. The r-CNN performs better in fine estimation of thickness but requires more effort to be trained. However, the c-CNN has lower precision, especially for thick samples.

Crystal structure governs material properties, and the carbon allotropes are typical examples. Even for carbon nanotubes (CNTs), the different structures that are determined by chiral indices show distinct electric properties. Furthermore, 2D materials with the same composition could have different electrical properties^{[105, 106]}. For example, the 2H phase MoS₂ is semiconductive and the 1T phase MoS₂ is metallic^{[107, 108]}. The diffraction mode of TEM is widely used to determine crystal structures, from the microscale to the nanoscale. The diffraction pattern that conforms to Bragg's law has a bright contrast. The positions of the diffraction spots with respect to the center transmission spot determine the crystal structure. HRTEM unveils atomic configuration and can also be used for structure studies, such as chiral indices.

The classification of crystal structures can benefit from the use of neural networks. Ziletti et al. adopted an automated method to classify the structure of a dataset of 3D materials based on their crystal symmetry^[62]. To skirt the much prior expertise that was traditionally required in classification algorithms, this research uses the ConvNet and the architecture that is shown in Fig. 8(a). ConvNet performs well in image recognition and can learn features in classification. The first step to analyze 3D structures in 2D images is to find a proper descriptor to represent the material. Diffraction patterns in x-axis, y-axis and z-axis overlapped in one image delegated by red, green and blue are introduced to represent the crystal structures^[62]. Note that the quadrature axis is arbitrary, and little is required to be known about the sample’s crystal symmetry. It is the crystal structure rather than the atomic composition that matters in the representation patterns, which makes this approach suitable for a good classifier because it differs the representations of diverse classes while reducing the difference between structures of the same class due to its robustness to defects. Given that the crystal structures have been transformed into more understandable diffraction fingerprints, 90% and 10% data are used for training and testing, respectively. Simulated crystal structures coupled with random displacements, substituting atoms and vacancies are combined to create an ideal dataset containing defective systems. Although the neural network itself in general figures out the features to be used in the task, it can be inferred that the classification is conducted based on the diffraction peak and their positions. As is shown in Figs. 8(c) and 8(d), the network shows robustness in tasks to categorize crystal structures, even in the case of structural transitions.

The properties of CNT are primarily determined by chiral indices^{[109, 110]}. In the analysis of chiral indices, the diameter and the chiral angle are the focus of research, which can be assisted by machine learning-based methods. Georg Daniel Forster et al. conducted the analysis task for chiral indices with the help of a neural network based on the classical LeNet-5, which proves to be effective on the HRTEM image of CNTs shown in Fig. 8(e)^[71]. Trained by simulated images with consideration of defects, this method was able to identify and classify the diameter of the nanotube by extracting of a contrast intensity profile (i.e., the distance between the dark and bright fringes). The chiral angle is measured through a Fourier Transform (FT). From the correlation matrix of the manual and automated results, especially in low diameter nanotubes, the machine learning-based method shows high accuracy in chiral structure determination of CNTs; as shown in Figs. 8(f) and 8(g).

Diffraction pattern identification is one of the most widely used operations during TEM characterization. The crystal structure determination algorithm can be extended for automatic diffraction pattern identification, which is of great significance when building an intelligent TEM operation system.

The local composition and the atom valence state are important information that influence the properties of a material. There are two methods for chemical analysis in TEM: energy-dispersive X-ray spectroscopy (EDX) and electron energy-loss spectroscopy (EELS). Both of them can be obtained in the STEM mode. The EDX is the characteristic X-ray signal that is emitted from the ionized atoms that are stimulated by the incident electron beam. EDX is collected in the STEM mode and has a nanoscale even atomic-scale resolution. The EELS signal results from the inelastic scattering interaction between the incident electron and atoms in the TEM sample.

EELS is a powerful tool to probe local chemical information with atomic resolution^[111-115]. The spatial environment of an electron in the TEM sample can be obtained by analyzing the energy loss of the diffracted electron beam^[116-118]. The analysis of massive generated EELS data is still challenging. To tackle this challenge, Roccapriore et al. presents two convolutional neural networks to establish a relationship between the NP geometries and the plasmon responses^[65]. In this way, spectral response can be determined by analyzing the structural imaging, which takes much less time to generate when compared to the spectral spectrum from the spectroscopy. In the Im2spec network, when the local subimages around specific locations (special descriptor) are input, the EEL spectra associated with the local configuration are presented; as shown in Fig. 9(a); and vice versa, in the spec2im network in Fig. 9(b), the spectra descriptors are considered as features, and the images of local configuration as targets are predicted by the network. In general, the mutual relationship between local configurations of NPs and the corresponding plasmon spectra diagram is established in the study. In many common cases, the result can be considered to be a basis in theoretical models. While for models that are not suitable to directly adopt this result, the establishment procedure is of referential significance. Although the obtained model is not suitable for the direct analysis of other TEM datasets with similar features, the network trained with this methodology could be applied as a pre-trained model for transfer learning in new TEM data analysis. Moreover, the pretreatment of training dataset and optimization of the network structure can improve the generalization ability of the model^{[119, 120]}.

EDX spectra can be obtained during STEM imaging with a high-spatial resolution for composition-related studies of nanomaterials. Due to the sensitivity of nanomaterials to the STEM electron beam, the acquired EDX spectra have to be limited and the scan speed has to be fast to reduce the electron-beam-induced defects. These experimental conditions result in a small number of obtained EDX spectra with high background noise, which is a considerable challenge for EDX spectra analysis. Han et al. offers two unsupervised deep networks to tackle these problems, a denoising network and a two-step neural network, which generate high-resolution 3D EDX spectra; as shown in Figs. 10(a)–10(c)^[67]. The two-step neural network first produces an intermediate noisy 3D reconstruction using the raw EDX projection data and then the projection data obtained from the noisy 3D reconstruction is enhanced using CNN. The consistency restriction is applied to guarantee that the calculated EDX spectra conform to the experimental data at the same angle. The two-step neural network can be applied to generate a large quantity of EDX spectra, even at the unmeasured angles. Because only the experimental data are used as the restrict condition, the neural network does not need a large number of labeled spectra for supervised training. Automatic EDX spectra processing and 3D reconstruction can be used in the chemical-property analysis of nanomaterials

In this paper, machine learning for in situ TEM is reviewed. Automated TEM data analysis has been summarized from the aspects of morphology, defect, structure, and spectra by machine learning. The merits of machine-learning methods for the corresponding data analysis tasks are discussed. Although the application of machine learning for TEM data analysis is a vibrant field and it is developing quickly, there are still several challenges, as follows:

(1) Natural and medical images have many datasets for network training and testing. However, there are few TEM datasets. Because of the data-driven feature, high quality and large quantities of TEM datasets are critical to achieve a trained network that has good performance. Image data augmentation of a manually annotated database could be a solution to this problem. Some studies have created simulated TEM images to meet the requirement of a much larger database with less labor. In addition, optimizing a network structure so that it requires less training data and still maintains performance could be an alternative solution to this problem.

(2) Machine-learning algorithms for TEM data analysis are usually optimized for a specific scenario. Models and network structure for TEM usually have so far been tailored for certain types of microscope images and materials. However, methods restricted in these conditions could lose robustness when adapted to new cases. Therefore, algorithms suitable for more general TEM data analysis need to be developed. At the present stage, the automation TEM data analysis is mainly focused on material. Electronic device characterization data obtained by in situ TEM and corresponding machine learning based data analysis (especially for the interfaces) are required.

(3) With the increased time and spatial resolution of in situ TEM characterization, the large amount of data that are generated in a short time create a challenge of data transfer and storage. Recently, compressing large amounts of data for electron microscopy by deep compressive sensing learning has been reported^[121].

(4) Most of the machine-learning methods adopted in analysis of TEM images are supervised. The unavoidable human bias of this method generates deviations in results, even from experts. Therefore, approaches that have fewer human interactions or unsupervised-algorithm aided methods should be developed for use in TEM.

This work is supported by NSFC under Grant Nos. 62074057, 62174056, Projects of Science and Technology Commission of Shanghai Municipality Grant Nos. (19ZR1473800, 18DZ2270800). ‘Shuguang Program’ supported by Shanghai Education Development Foundation and Shanghai Municipal Education Commission, and the Fundamental Research Funds for the Central Universities.