In order to solve the shortcomings of poor recovery effect and timeliness of current rain removal algorithms, this paper proposes a fast video rain removal algorithm that combines temporal gradient and horizontal direction gradients. First, the rain image is divided into a background layer and a rain race layer, and the model function is constructed by using the time direction gradient of background layer, the horizontal direction gradient of rain trace layer, and the corresponding pre-trained Gaussian mixture model of each layer as priori conditions; second, the semi quadratic splitting method is used to transform the non-convex function to obtain the objective function for minimization; finally, fast total variation method is used to optimize the model function to obtain the optimal solution of the model function. Experimental results show that, compared with other algorithms, the algorithm has better rain removal recovery effect, good timeliness and faster speed, and the average peak signal-to-noise ratio is improved by more than 1 dB.