The rapid development of wireless communication technology has made the non-regeneratable Radio Frequency (RF) spectrum resources extremely scarce. The In-band Full-duplex (IBFD) wireless communication scheme utilizes the same frequency to transmit and receive signals at the same time slot, overcoming the shortcomings of traditional half-duplex communication schemes that can only transmit and receive signals in different frequency carriers or different time slots, and doubling the utilization rate of the RF spectrum. However, RF self-interference is the primary problem that must be resolved for the application of IBFD scheme. Microwave photonic RF Self-interference Cancellation (SIC) technology is attracting more and more attention by virtue of the advantages of the large signal processing bandwidth and high amplitude and time regulation accuracy. In this paper, considering the influence of environmental variation on the RF SIC performance, we focus on the control algorithm for the microwave photonic RF SCI system.Firstly, a theoretical model of microwave photonic RF SIC link is established, and the expressions of RF SIC value for the single-frequency signal and the signal with a certain bandwidth are derived. Then, the parameters that affect the RF SIC performance are analyzed, including the amplitude mismatch and delay mismatch. The analysis results provide a basic reference for the design of the microwave photonic RF SIC scheme, the optimization of the regulation units in optical domain and the construction of the experimental system.Secondly, an adaptive feedback control system for microwave photonic RF SIC system is designed and constructed, which is composed of the optical domain regulation units, RF signal down conversion units, data acquisition units and data processing units. The Field Programmable Gate Array (FPGA) is applied for digital signal processing, adaptive algorithm iteration and control instruction output to the regulation units in optical domain via serial communication unit. Also the clock conversion is conducted in FPGA to provide different clock for different digital unit.Thirdly, an adaptive algorithm for microwave photonic RF SIC system in FPGA is implemented. The algorithm is divided into two steps. One step is a cross-correlation algorithm to obtain the amplitude and delay mismatch between the interference signal and the reference signal, which provides the initial value for the other step of Particle Swarm Optimization (PSO) algorithm. The PSO algorithm realizes a further RF SIC through the circle of the acquisition of the residual self-interference power by Analog-to-digital Converter (ADC), the optimization iteration for control instruction to the regulation units in optical domain. During the circle, the tuning accuracy of regulation units in optical domain and the sampling accuracy of ADC are considered comprehensively to optimize the iteration logic loop, which are applicable for the really established system.Finally, a microwave photonic RF SIC system with direct modulation is established, for which the feedback control algorithm based on FPGA is demonstrated. The cancellation depth of 35 dB for the 2.4 GHz center frequency and 40 MHz bandwidth is realized by the microwave photonic RF SIC system. The measured results verify the feasibility and efficiency of the FPGA based control algorithm.