Accompanying the demands for laser communication network integration, lightweighting, and the widespread adoption of transceiver integration, achieving efficient isolation between optical antenna transmission and reception is crucial. This article thus investigates the isolation between coaxial and off-axis antennas. Initially, using Code V optical design software, coaxial antennas and off-axis two-reflective antennas operating in the 1 550 nm band were individually designed with a field of view angle of ±1.5 mrad, approaching the diffraction limit. Subsequently, based on the stray light scattering model, ray tracing was conducted using analysis software to simulate isolation. Methods were then studied to place baffles at the center of coaxial antenna secondary mirrors to mitigate aperture risk, and to adjust off-axis tilt and curvature radii in off-axis antennas. Ultimately, the isolation levels of coaxial and off-axis antennas were respectively increased to -69 dB and -89 dB. The results indicate that the aforementioned methods effectively enhance the isolation of optical antennas.