The ViBe algorithm has been widely used for moving target detection owing to advantages such as ease of realization and high operational efficiency. However, the algorithm has weaknesses concerning ghosts, shadows, and incomplete moving targets. Therefore, an improved ViBe algorithm combined with the average background method is proposed, which aims to address the ghost phenomenon existing in the foreground detection of the ViBe algorithm, and the difficulty regarding its long-term elimination. First, the improved average background method was used to obtain and initialize the true background. Then, the ViBe algorithm was used for both foreground detection and background updating, to eliminate the ghosts in subsequent frame detection. Finally, morphological knowledge was used to make the target more complete by eliminating holes and interference targets. The experimental results indicate that this algorithm can more effectively eliminate ghosts and improve the detection accuracy compared with the traditional ViBe algorithm.