UAV image matching occupies a central position in UAV image processing, and orientation estimation is an important part of performing rotation invariant matching. However, due to the inability to specify the standard orientations of the UAV image feature points, the current matching method still suffers from large orientation estimation errors, resulting in low matching accuracy. A Rotation Invariant Matching Network (RIMN) based on multi-scale feature fusion is proposed for UAV images, in which the multi-scale featureextraction module is used to aggregate rich semantic features of the images and the Transformer self-attention block is used to extract robust features in the weakly textured regions of the image. Meanwhile, a double constrained loss function is designed to improve the orientation estimation accuracy of the feature points. Finally, image matching comparison experiments under different rotation angles is set up. The qualitative and quantitative results show that this method has better rotation invariant matching performance.