A sub-mirror support optimization model based on the outer envelope of the traditional secondary mirror support structure was established, according to the characteristics of the traditional coaxial reflective secondary mirror support structure. The variable density topology optimization method was used to optimize the model. According to the topology optimization results, a coaxial reflective secondary mirror support structure with an occlusion ratio of 0.085 was designed. The finite element simulation results show that the basic frequency and structural stiffness of the coaxial reflective secondary mirror support structure are significantly improved compared with the traditional three-wing and four-wing structures in the case of the same structure shielding ratio and mass. The optimized secondary mirror support structure was processed and a random vibration test was conducted. The test results show that the optimized basic frequency of the secondary mirror support structure is 470 Hz, the finite element simulation result is 477 Hz, and the relative error is within 2%, which verifies the reliability and authenticity of the finite element calculation model. The secondary mirror support structure is suitable for use in a coaxial reflective optical system that requires a lower mask ratio and higher stiffness.