The scanning rotation mirror is one of the most important components in laser scanning detecting system. Based on the principle of geometrical optics, an analytical expression for the scan trace of hexahedral column rotation mirror is derived. The relationship between the position of incident beam to rotation mirror and the scan trace is also analyzed. In addition, the relation curve of scan trace versus scanning angle is produced. The structural characteristics of the system are the origin of creating scanning nonlinearity and asymmetry. In order to reduce the nonlinearity of scan trace, a curved rotating mirror scanning system is proposed, and the scan trace for curved rotating mirror is computed. The numerical results for the designed rotation mirror system show that the system can solve the nonlinearity problem of plane rotating mirror effectively.