The number of correction modes, an important parameter for the modal control of an adaptive optics system, has a significant effect on the system performance. The mode space of aberrations is built up by the singular value decomposition of the system′s response function matrix. The number of correction modes is determined by the root mean square estimation of the residual aberration obtained under different numbers of correction modes. A dynamic optimization method for modal control of adaptive optics system is proposed. An adaptive optical experimental system mainly based on a Hartmann-Shack wavefront sensor and a membrane deformable mirror is constructed. The above control method is verified by fitting different aberrations. Results show that the residual aberration with correction by the dynamic optimization method of modal control is smaller than the one with correction by the method of the fixed mode number, and the spatial fitting performance of the adaptive optics system can be obviously improved.