Fluorescence microscopy has become an essential tool for biologists, to visualize the dynamics of intracellular structures with specific labeling. Quantitatively measuring the dynamics of moving objects inside the cell is pivotal for understanding of the underlying regulatory mechanism. Protein-containing vesicles are involved in various biological processes such as material transportation, organelle interaction, and hormonal regulation, whose dynamic characteristics are significant to disease diagnosis and drug screening. Although some algorithms have been developed for vesicle tracking, most of them have limited performance when dealing with images with low resolution, poor signal-to-noise ratio (SNR) and complicated motion. Here, we proposed a novel deep learning-based method for intracellular vesicle tracking. We trained the U-Net for vesicle localization and motion classification, with demonstrates great performance in both simulated datasets and real biological samples. By combination with fan-shaped tracker (FsT) we have previously developed, this hybrid new algorithm significantly improved the performance of particle tracking with the function of subsequently automated vesicle motion classification. Furthermore, its performance was further demonstrated in analyzing with vesicle dynamics in different temperature, which achieved reasonable outcomes. Thus, we anticipate that this novel method would have vast applications in analyzing the vesicle dynamics in living cells.Fluorescence microscopy has become an essential tool for biologists, to visualize the dynamics of intracellular structures with specific labeling. Quantitatively measuring the dynamics of moving objects inside the cell is pivotal for understanding of the underlying regulatory mechanism. Protein-containing vesicles are involved in various biological processes such as material transportation, organelle interaction, and hormonal regulation, whose dynamic characteristics are significant to disease diagnosis and drug screening. Although some algorithms have been developed for vesicle tracking, most of them have limited performance when dealing with images with low resolution, poor signal-to-noise ratio (SNR) and complicated motion. Here, we proposed a novel deep learning-based method for intracellular vesicle tracking. We trained the U-Net for vesicle localization and motion classification, with demonstrates great performance in both simulated datasets and real biological samples. By combination with fan-shaped tracker (FsT) we have previously developed, this hybrid new algorithm significantly improved the performance of particle tracking with the function of subsequently automated vesicle motion classification. Furthermore, its performance was further demonstrated in analyzing with vesicle dynamics in different temperature, which achieved reasonable outcomes. Thus, we anticipate that this novel method would have vast applications in analyzing the vesicle dynamics in living cells.