The multi-sensor information fusion method of the existing multi-object tracking algorithms for self-driving cannot give full play to synergy. To solve this problem, a three-dimensional multi-object tracking algorithm based on multi-modal feature fusion and learnable object similarity estimation is proposed. The multi-modal feature fusion module fuses the feature of images and point clouds on the basis of the channel attention mechanism to further improve the expressive ability of multi-modal features. The object similarity estimation module directly generates the similarity matrix through the network, and realizes the cross-modal joint reasoning between multiple objects in a learnable way, which avoids massive manual parameter setting. The proposed algorithm is verified and tested on the KITTI data set, and its higher-order tracking accuracy (HOTA) reaches 69.24% in the test set, which indicates that the algorithm is superior to other algorithms in accuracy and has good robustness.