A geo-localization method is proposed for military and civilian applications, which is used when no global navigation satellite system (GNSS) information is available. The open graphics library (OpenGL) is used to build a three-dimensional geographic model of the test area using digital elevation model (DEM) data, and the skyline can thus be extracted with the model to form a database. Then, MultiSkip DeepLab (MS-DeepLab), a fully convolutional semantic segmentation network with multiple skip structures, is proposed to extract the skyline from the query image. Finally, a matching model based on convolutional neural network (CNN) feature is adopted to calculate the similarity between the skyline features of the query image and the DEM database to realize automatic geo-localization. The experiments are conducted at a 202.6 km2 test site in north-eastern Changsha, China. 50 test points are selected to verify the effectiveness of the proposed method, and an excellent result with an average positioning error of 49.29 m is obtained.