To improve the real-time performance,the accuracy and the robustness of the unmanned aerial vehicle (UAV) autonomous navigation system under time-varying noise circumstance, an integrated navigation system of machine vision and inertial guidance was established, and an adaptive square-root unscented Kalamn filter (ASRUKF) was proposed. By introducing the Mahalanobis distance of the residual between the observed and predicted values to the minimal skew sampling square-root Kalman filter, the new algorithm can restrain the system noise adaptively, compute faster and be more robust to noise. The simulation results show that compared with the SRUKF, the ASRUKF is more robust to noise, and the computation time is reduced by about 38.8%, the forecast accuracy of displacement, velocity and attitude angle increases by more than 4 times, 4times, 6 times, respectively.