When a structure material is damaged by impact events, the reliability and lifetime of the material will be severely affected. So impact location is considered as the prime approach for structural health and damage monitoring. In this study, a novel fiber Bragg grating (FBG) impact location system based on broadband light source is designed, aiming at the shortcoming of existing location systems based on FBG. An improved localization algorithm based on the time difference of arrival (TDoA) is proposed for impact location. According to this algorithm, the impact position can be accurately predicted without wave velocity. Impact planar location experiments are carried out for verification of the FBG impact location system and algorithm on a 400 mm×400 mm×3 mm aluminum alloy plate. The resulted locating error shows high precision and good stability of the proposed system.