In order to improve the control precision of the three-axis operating system used in airborne laser pointer, a simulating model is established for system analysis. A three-axis, coupling and nonlinear dynamic model is built based on Lagrange theorem, and by analyzing the velocity coupling and the dynamics coupling relationships among the axis of frames, their analytical relationships in theory are deduced. Then, according to electrics and mechanic equations of the actuator-torque motor, the open-loop transfer function matrix of the control variable is established, and a cascade compensator is designed to decouple the control system. Analysis results show that the system can satisfy the quick-response requirement at working in stablization after decoupling and adding correction function in series. The step response time are 0.45 s for the orientation frame, 0.50 s for the pitching frame, and 0.85 s for the across frame. Finally, the electro-mechanical model is verified through computer simulation, which sets a foundation and provides theoretical references for further research and design of the control system of high-precision airborne laser pointer.