Orbital angular momentum (OAM)-carrying beams have received extensive attention due to their high-dimensional characteristics in the context of free-space optical communication. However, accurate OAM mode recognition still suffers from reference misalignment of lateral displacement, beam waist size, and initial phase. Here we propose a deep-learning method to exquisitely recognize OAM modes under misalignment by using an alignment-free fractal multipoint interferometer. Our experiments achieve 98.35% recognizing accuracy when strong misalignment is added to hyperfine OAM modes whose Bures distance is 0.01. The maximum lateral displacement we added with respect to the perfectly on-axis beam is about ±0.5 beam waist size. This work offers a superstable proposal for OAM mode recognition in the application of free-space optical communication and allows an increase of the communication capacity.