Considering that the fully-convolutional siamese network algorithm for object tracking (Siamfc) algorithm is prone to tracking failure in cases such as heavy occlusion, rotation, illumination variation, scale variation, a siamese neural-network object-tracking algorithm with the distractor-aware model is proposed. First, the low-layer structural and high-layer semantic features were extracted from siamese networks; then, they were effectively fused to improve the representation ability of the feature. Second, the template adaptive strategy was used to update the template online to improve tracking accuracy in cases of occlusion and rotation. Simultaneously, the distractor-aware model based on color histogram features was introduced into the algorithm. The target response map was obtained by weighted fusion to estimate the position of the target while the adjacent frame scale adaptive strategy was used to estimate the optimal scale. To verify the effectiveness of the proposed algorithm, its performance was compared with those of various tracking methods on open-source datasets. Experimental results on the standard test dataset of the 2015 ^th object tracking show that the overall tracking accuracy and success rate of the proposed algorithm are 0.945 and 0.929, which is 2.9% and 2.8% higher than those of the Siamfc algorithm, respectively. Further, the proposed algorithm performs with high accuracy and success rate in the aerial test dataset of an unmanned aerial vehicle (UAV).