An omnidirectional staring multi-target tracking and processing system based on infrared fish-eye lens is proposed to satisfy urgent requirements of large-viewing-field infrared multi-target real-time reconnaissance and tracking. Utilizing infrared fisheye lens and 512 pixel×512 pixel PtSi thermal detector modules, the system can detect threat targets with the whole space-domain capacity and the whole time-domain real-time capability. The hardware of the infrared multi-target processing system based on dual-digital signal processor system is designed to implement data processing for multi-target recognition and tracking algorithms. The read-out circuit and the logical block diagram are presented. The multi-target tracking experiment for some motor targets is carried out, and infrared image sequences including targets 1 and 2 are acquired. Theoretical and experimental tracks of the targets are compared, and availability of the system and real-time capability of multi-target tracking are validated.