Aiming at small IR target dectection and tracking in complex background, a method based on the top-hat detection algorithm, genetic algorithm and particle filter is presented. Firstly, a sub-frame extracting method is used for the removal of the background and noise around target, which effectively reduces the number of pixels participating in operation; secondly, a few structures with different edge characteristics are introduced in the top-hat algorithm, which greatly improve the efficient for the detection of the pre-target from sub-frame; thirdly, based on the correlation of target movement in time and space domain, a threshold judgment technology is used to remove the false targets; finally, the genetic algorithm is introduced into the particle filtering to enhance the diversity of particles, which can effectively improve the real-time and the precision of target tracking. The experimental results show that the presented algorithm has obvious superiority in detection correctness and tracking accuracy and robustness.