To address the problems of low matching accuracy, high computational cost and slow convergence speed of point cloud registration methods in Euclidean space, a point cloud registration algorithm based on geometric algebra was proposed by using geometric algebra’s expressive power for high dimensional space. Firstly, the point cloud data was transformed into geometric algebraic form, and based on the rotor of geometric algebra, the cost function of point cloud registration in geometric algebra space was given. Secondly, combined with the normalized least mean square algorithm, the solution of the rotor was simulated as a signal filtering problem, and the rotor iteration formula was constructed based on the steepest descent method in the geometric algebraic space, so that only one points pair instead of all point pairs was used for each calculation. The rotor obtained by iterative calculation could be used for any dimensional rotation estimation problem, so that the three-dimensional point cloud was gradually rotated and registered. Finally, in order to further optimize the conflict between the convergence speed and the steady-state error, a variable-step rotor iteration formula was given by using the Sigmoid function, which can speed up the convergence speed while reducing the steady-state error. The registration performance of the proposed algorithm was verified by using the model data set and the public data set. Compared with the classical iterative closest point algorithm, the registration accuracy of the model data set is increased from 10^-2 to 10^-8 orders of magnitude, and the registration accuracy of the public data set is increased by 35%. The proposed algorithm has faster convergence speed, higher registration accuracy and lower steady-state error.