To guarantee the high stability and high precision of an off-axis thee-mirror optical system space camera, a integration front frame structure was designed to support the second mirror and folded mirror according to the characteristics of the same height of second mirror and folded mirror in optical axis direction, and a topology optimization method based on constraint mode and free mode was proposed to optimize the front frame structure. Then, the integrated structure was assembled into the entire camera after optimization, and the finite element analysis of static was carried out. The results show that the camera maintains excellent static performance with the optical tilt between the primary mirror and secondary mirror being less than 9″ and the optical tilt between the primary mirror and fold mirror being less than 22.4″, the optical eccentricity between the primary mirror and secondary mirror being less than 0.021 mm, meeting the tolerance requirement of system. By free modal analysis and test to the integrated front frame structure, the results verify the correctness of the design method. The proposed topology optimization method can efficient avoid the defect of topology optimization based on constraint modal frequency that there is no relationship between the constraint points. It can provide reference for design the space camera with high-resolution and wide field.