A new sliding mode controller is designed in order to compensate nonlinear uncertainties of the servo system and reject high frequency chattering phenomena of sliding mode control. At first, a third-order Extended State Observer (ESO) with an inverse hyperbolic sine function is presented. The ESO can estimate state variables and uncertainties of a second-order system. Then, a sliding mode control law is constructed with the inverse hyperbolic sine function, and a sliding mode controller with the ESO is designed. The asymptotic stability of the sliding control mode system is proved. Finally, the simulation experiments of the three-axis turntable servo system demonstrate the ESO could obtain the higher precision estimations of state variables, and the servo system with the new controller has the higher position tracking and velocity tracking precision. Moreover, the sliding mode control input signals reduce high frequency chattering phenomena.