In order to solve the problems of image differences in scale, rotation, grayscale and 3D viewing angle, and achieve adaptive target initialization during electro-optical imaging terminal guidance, a new scene matching framework based on random Ferns classifier was constructed. The process of realizing the method includes classifier off-line training which yields fast run-time performance was performed. Candidate match regions between reference image and run-time image were found by the classifier. Scale invariant feature transform (SIFT) descriptors of corresponding regions in each candidate matches were computed and false matches feature pairs rejecting was performed based on Mahalanobis distance criterion. Epipolar geometry of the two images was estimated by applying PROSAC to the central locations of the corresponding regions in the final matches. Target location and size in run-time image were computed based on the epipolar geometry. Simulation results show that the proposed method provides robust target initialization during electro-optical imaging terminal guidance and is more stable than original Ferns methods under severe conditions.