The relay mirror technology is an important system combat concept under extensive research. Beam blocked and truncated by the receiving Cassegrain telescope causes serious power losses in beam uplink propagation in the relay mirror system, which degrades performance of the relay mirror system. Using vortex beam source and phase optimization is an effective way to improve power efficiency of beam uplink propagation in the relay mirror system. A reduced-scale experimental installation is established with the same Fresnel number as beam uplink propagation process in the relay mirror system with the following parameters as “1.2 m source aperture, 30 km uplink propagation distance, receiving telescope with 1.2 m outer aperture and 0.24 m inner aperture”. Vortex source is generated by using the NdYVO4 laser and a liquid crystal spatial light modulator with spiral phase distribution, and phase distribution of the vortex source is optimized by using the stochastic parallel gradient descent algorithm. Reduced-scale experimental study on beam uplink propagation in a relay mirror system with vortex source and phase optimization is performed. Results show that power efficiency of beam uplink propagation in the relay mirror system is improved from 71.89% to 91.59% by using vortex source and phase optimization.