The infrared polarization object tracking under dynamic scenes has a demand for real-time performance and low power consumption. FPGA has the characteristics of parallel computing, which can greatly improve the system throughput and data processing speed, and can meet the requirements of real-time. Therefore, a target detection and tracking system based on FPGA was designed and implemented. On the hardware development platform, the methods of modularization and software and hardware collaborative design were adopted to realize the tasks with different computing characteristics in PS (ARM) and PL (FPGA). PL was responsible for the acceleration of some algorithms, data transmission between FPGA and ARM processors, HDMI and other interface logic control. PS was responsible for the implementation of more complex detection and tracking algorithms, and controlled each module in FPGA. Finally, the target detection and tracking system was tested and analyzed, and the hardware resource consumption and power consumption of the system were also given in the experiment. The results showed that the purposed target detection and tracking system can meet the requirements of real-time and low power consumption..