Aiming at the problems of tracking failure and model degradation caused by low utilization rate of polarization factor, complex background and drastic change of target scale and appearance in traditional infrared ship target tracking scene, a bidirectional correlation filtering target tracking algorithm based on joint optimization framework is proposed. On the basis of constructing a new polarization constraint self-adaptation filter-location joint optimization framework, the response map of the main filter is calibrated by polarization mask filter, a parallel double filter structure based on polarization characteristics is constructed, and the convergence of the proposed optimization framework is proved. In addition, a dataset of infrared dim ship target is constructed, and experiments on the dataset prove that the proposed algorithm can realize real-time tracking, and it is more than 15% higher than other related filtering algorithms recently proposed in terms of accuracy, robustness and other common performance indexes.