The hysteresis nonlinearity of piezoelectric actuator is one of the main defects in the control of deformable mirror which is widely used as a key component in adaptive optics system. This letter put forward a modified Prandtl-Ishlinskii (PI) model in order to precisely describe the hysteresis nonlinearity of piezoelectric actuator. With this proposed model, an inverse-model based controller used for trajectory tracking in open-loop operation is designed to compensate the hysteresis nonlinearity effect. Then, some tracking control experiments for the desired triangle trajectory are performed. From the experimental results, we can see that the positioning precision in open loop operation is significantly improved with this inverse-model based controller.