The beam adjustment method has been widely used in oblique aerial image orientation tasks as a traditional image pose calculation method. However, it highly depends on a good initial value. This dependence often results in slow convergence or convergence failure in flight missions with missing navigation systems or limited navigation accuracy. Therefore, to effectively iterate when the initial value is missing or of poor quality, an oblique image orientation method based on a local-to-global optimization strategy is proposed in this paper. This strategy improves the robustness of the optimization iterative solution. In the proposed method, first, a “local map” (consisting of a nadir and four oblique images) according to the maximum overlap principle is constructed, and then, a “global map” is formed by optimizing these local maps. In the local map, the point of the image with the same name is used as the observation value of the constrained optimization to obtain the unknown parameters and corresponding weight matrix (information matrix) of each local map. In the global map, the estimates of variables and the corresponding weight matrix of the local map constitute the observations and weights of the global optimization problem. The proposed method was tested using a set of large simulated dataset and a set of small, real international public test dataset. The experimental results show that compared with the traditional methods, the proposed method can converge faster and more stably under the premise of no loss of accuracy.