Off-axis reflector telescopes are mainly used in space astronomy observation and other related fields. The image quality of off-axis two-inversion telescopes is sensitive to lens misalignment and is more difficult to calibrate using a laser interferometer after misalignment in an operating environment. To address this challenge, this paper proposes a method that uses the out-of-focus spot map of the system for infinity point targets and uses the Swin-Transformer network to calculate the amount of lateral misalignment of the secondary mirror. Theoretical calculations were used to determine the camera defocus positions that could avoid the multi-solution problem, and simulations were used to investigate the effect of different defocus amounts on calibration accuracy. The trained network uses a frame of out-of-focus spot map of the out-of-tune system to perform the estimation of the amount of out-of-tune. Both simulation analysis and experimental results verify the effectiveness of the method, which can achieve high accuracy and fast correction of out-of-tune telescope systems in the working environment.