Surface plasmons have novel optical properties, such as breaking light diffraction limit, surface localization, and optical near field enhancement, so they have been widely applied in the research area of photovoltaics, photocatalysis and photodetectors. Surface plasmons have excellent light harvesting capability, which can enhance the efficiency of conventional semiconductor devices by integrating with the conventional semiconductor devices. The hot electrons generated from the decay of localized surface plasmons is the core element in converting the incident light to the electrical or chemical energy. Therefore, the study of plasmonic hot electrons generation and relaxation process in microcosm is essential for the design of high-efficiency plasmonic nanophotonic devices. This article reviews the relaxation process of surface plasmons and recent progress in the ultrafast dynamics of plasmonic hot electrons in metal-semiconductor interface, discusses the remained issues and prospects future application of plasmonic hot electrons.