In view of the problems of small size and blurry appearance of images shot from the perspective of UAV, and motion of the UAV platform itself, the detecting and tracking of moving objects based on UAV images is still very challenging. A moving object detecting and tracking algorithm based on the improved LK optical flow method is proposed, and the Kalman filter algorithm is used for object tracking. In order to solve the problem that the traditional LK optical flow method is not effective in the dynamic environment, the semantic segmentation thread is introduced to screen the dynamic points and static points, the motion compensation of the UAV is completed by solving the homography matrix through the static points, and the optical flow value of the dynamic points are calculated to obtain the optical flow points for reducing the mismatch rate. Finally, the complete moving object is obtained through the clustering algorithm and morphological algorithm, and the proposed re-identification module is introduced into the Kalman filter algorithm to realize object tracking. The experimental results verify that the moving object detecting and tracking algorithm designed here can accurately extract the moving object in real time from the perspective of the UAV, and track it continuously and stably.