Fast steering mirror with superior positioning accuracy and speed are essential for enhancing secondary image stabilization, which is critical for high-resolution imaging. They help in mitigating the effects of low-frequency vibrations. The continuous advancements in detectors, CPUs, and image processing algorithms have significantly improved the systems' ability to handle higher frame rates and detect image shifts with greater precision. As a result, composite axis optical systems that incorporate these technologies are becoming more adept at reducing image shifts caused by medium and high-frequency vibrations. This progress opens up extensive potential for applications across various fields that require high-precision imaging capabilities. This paper delves into the intricate design of a coaxial common optical path tracking and imaging composite axis optical system, which employs a Fast Steering Mirror (FSM) for secondary image stabilization. The FSM, acting as the actuator responsible for compensating image shifts, benefits from a smaller aperture, which translates into a significant enhancement in its response speed and closed-loop bandwidth. This feature is particularly advantageous in scenarios where rapid and precise adjustments are required to counteract image shifts. In the quest for the optimal front group configuration for the composite axis optical system, the paper conducts a thorough analysis of reflective and catadioptric afocal systems, weighing their respective pros and cons. The selection process culminates in the choice of a catadioptric afocal system, which is capable of achieving a larger beam expansion ratio. This system is particularly well-suited for the front group of the composite axis optical system, given its ability to meet the stringent small aperture requirement of the FSM in the optical path. The catadioptric afocal system is designed to provide a long focal length through a more compact Cassegrain configuration, while the short focal length component is delivered by a transmission system. To address the chromatic aberrations that may be introduced by the lenses, a double cemented lens is employed. Although this approach adds complexity to the optical group compared to a two-mirror system, it becomes a viable solution when the distribution of long and short focal lengths is carefully managed, allowing for a substantial beam expansion ratio. Following the selection of the appropriate front group structure, the paper employs Zemax software to validate the catadioptric afocal optical system. The validation process involves an in-depth analysis of the point spread function and optical distortion, which are obtained by placing a near-axis plane behind the afocal system. The parallel nature of the light emitted by the front group ensures that the defocusing effect caused by the rotation of the FSM remains within an acceptable range, thus minimizing its impact on image quality. Building on the understanding of the mechanism behind image rotation generation, the paper constructs a mathematical model that correlates image rotation with the Modulation Transfer Function (MTF). MATLAB is then utilized to simulate and analyze the impact of image rotation at various detector positions on the MTF, particularly when the FSM compensates for linear image shifts. This analysis identifies the position at the edge of the field of view where imaging quality is most susceptible to the effects of image rotation. By establishing the relationship between the transfer function and the rear group imaging focal length at the position most affected by image rotation, the paper explores the optimal rear group focal length that would yield the best modulation transfer function across different spatial frequencies. The results demonstrate that the impact of image rotation on image quality can be effectively reduced to an acceptable range, thus validating the theoretical feasibility of achieving secondary image stabilization through the use of a fast steering mirror. In conclusion, this paper not only underscores the theoretical viability of employing fast steering mirrors for secondary image stabilization but also for the design and development of composite axis optical systems that leverage the advantages of small aperture fast steering mirrors.