In recent years, photoelectric tracking technology has developed rapidly and has been widely used in space laser communication, laser weapons and other fields. Traditional optoelectronic tracking systems mostly use servo turntables or optical mirrors to achieve beam adjustment to ensure accurate tracking of the target. However, they usually have a large size and weight, and excessive sensitivity to vibration also affects their dynamic tracking performance. A typical representative of a small-inertia beam servo system is the rotating double prism, which consists of a pair of circular optical prisms with specific wedge angles. Both prisms rotate with the same central axis to achieve fast beam deflection, which has the advantages of high accuracy, low rotational inertia, fast beam pointing, and low vibration sensitivity, which makes the double prism system of important practical value in laser communication, interferometry, photoelectric detection, and other fields. The key to achieving target tracking using double prisms is to reveal the nonlinear variation mechanism of beam delivery and to develop an effective prism control strategy. In this paper, a beam compound tracking technology based on a rotating double prism is proposed to replace the traditional servo turntable to achieve precise optical tracking. Firstly, the beam deflection model of the double prism is established, the relationship between the beam deflection vector and the prism angle is deduced in detail, and the nonlinear problem of the prism rotation in the tracking process is analyzed. A double-prism composite tracking system is designed to change the direction of the beam by using two independently rotating prisms, each of which is rotated under the drive of a motor. In the double-prism tracking system, an improved controller is designed to improve the tracking performance, and an FSM is added behind the double prisms, and the beam passing through the prisms will be deflected by the FSM for the final error complement. Among them, the rotating double prism is responsible for a wide range of beam adjustment, and a large field-of-view target viewing camera is added to the system for searching the target position and initial alignment of the optical axis. The FSM is used to compensate for the beam control error of the double prism, and the tracking camera is used as the target detection element in the system to obtain the aiming error of the optical axis in real time. The inertial measurement unit is mounted on an optical abutment. Unlike traditional two-dimensional servo turntable, the rotating dual prism system has a special structure and beam deflection mechanism, which cannot achieve optical axis stabilization through on-axis gyroscopic feedback as in traditional two-dimensional servo turntable, so the dynamic compensation of the optical axis is achieved through attitude decoupling. An experimental system is built to verify the compound tracking technology of the rotating double prism. A collimator is used in the direction along the center axis of the rotating double prism to generate near-parallel light to simulate a long-range target, and the rotating double prism system is fixed to a six-degree-of-freedom swing table for simulating attitude perturbations of a moving pedestal platform. In the dynamic tracking experiment, the control accuracy of the biprism using the improved controller is obviously improved. Compared with the PID controller and the linear active disturbance rejection controller, the control accuracy of the prism is increased by 58.33% and 32.81% respectively, and the corresponding tracking errors are reduced from 49.03 μrad and 38.88 μrad to 31.15 μrad. After turning on boresight compensation, the tracking performance is further improved, with the total tracking error reduced to 7.49 μrad and the tracking accuracy increased by 4.16 times. The experimental results show that the beam compound control can effectively improve the tracking accuracy of the rotating double prism, which verifies the effectiveness of the proposed method.