In order to improve the measurement accuracy of Micro Electro Mechanical System (MEMS) gyroscope, a time varying Auto-Regressive and Moving Average (ARMA) Model compensation method based on forgetting factor recursive least squares estimation was proposed. According to the measured MEMS gyro random drift signal without trend item, the stability was analyzed by the subsection test and the time varying ARMA was built with the suitable basis function and subspace dimension. The model parameters were estimated with the Forgetting Factor Recursive Least Square (FFRLS) method by setting forgetting factor to make it possible that the model parameters can reflect the dynamic change of the signal. For the time varying parameters with slight fluctuation, the 5 order polynomial was used to fit the parameters of the time-varying model, and an analytical method was proposed to optimize the parameters, so as to establish the optimal random drift model. And the modeling results were applied to Kalman filter for random drift compensation. The compensation results of the proposed method were compared with the compensation results of the time invariant ARMA modeling compensation method. The comparison results indicated that the variance of the signal with the proposed method compensation is nearly 40% reduced by the variance of the signal with the time invariant ARMA model compensation. So the compensation precision of MEMS gyro random drift was improved effectively.