In the design of a two-degree-of-freedom fast steering mirror system, in order to increase the control bandwidth of the system, the low-order natural frequency in the working direction should be reduced as much as possible, and the high-order natural frequency in the non-working direction should be increased. This subject used a deep-cut flexure hinge fast-reflection mirror system as the research object. First, the vibration mode movement direction of the first four-order natural frequency of the system was analyzed, and considering that the traditional stiffness calculation method was not suitable for the third-order mode direction problem, the stiffness calculation formula in the third-order mode shape direction was re-derived; secondly, the working stiffness of the deep-cut flexure hinge was deduced by the energy method and the second card theorem, and the nonlinear fitting was simplified. The error between the simplified calculation formula calculation result and the finite element simulation result did not exceed 8.9%, which proves the accuracy of the derived hinge working stiffness theoretical formula; then, the third-order mode shape direction stiffness calculation formula and the flexible hinge stiffness calculation formula were substituted into natural frequency calculation formula and finite element verification. The results showed that the error between the theoretical formula calculation result and the finite element simulation result did not exceed 1.7%, which proves the accuracy of the new third-order mode shape direction stiffness calculation formula. Finally, using genetic algorithm, multi-objective optimization design was carried out on the first four-order natural frequency of the system, and the design requirements were reached. The optimized structure obtained was significantly optimized compared with the initial structure, the stiffness in the working direction was reduced by 19.04%, and the stiffness in the non-working direction was increased by 297.83% and 77.09%. In addition, it has been verified by finite element simulation, and the results showed that the first and second order fundamental frequencies were reduced by 8.08% and 5.40%. The third and fourth-order fundamental frequencies have increased by 112.59% and 16.80%. It proves that the optimized structure is greater than the initial structure, which can effectively increase the system control bandwidth.