A real-time target tracking algorithm based on improved SiamFC (a fully convolutional Siamese network) is proposed herein to improve the target tracking speed and the network discrimination ability of traditional SiamFC. First, after the first conventional convolution layer, a depth-wise separable convolution is set to improve the tracking speed by reducing the amount of parameter calculation. Second, the third convolution layer is set as a mixed depth-wise convolution (MixConv) to improve the recognition ability of the network. We extracted features from the convolution kernel of different sizes and concatenated them in the channel to achieve a multifeature fusion and extract more robust features. Finally, the preprocessed ILSVRC2015 data set was used to train the network using the random gradient descent method, and the performance of the algorithm was tested on the OTB2015, VOT2016, and ILSVRC2015 data sets. Experimental results show that compared with the SiamFC algorithm, our algorithm shows a certain improvement in the tracking success rate, tracking accuracy, and tracking speed, and can meet real-time tracking requirements.