Based on a compact two-dimensional finite difference frequency domain (2D-FDFD) method with anisotropic perfectly matched layer absorbing boundary conditions, an octagonal photonic crystal fiber (O-PCF) is analyzed and investigated. Through computing the first twenty modes of octagonal photonic crystal fiber, we find that the circular field distribution of the octagonal photonic crystal fiber is better than that of hexagonal photonic crystal fiber (H-PCF). The phase diagrams of cut-off, single mode and multi-mode for the two fibers are obtained through analyzing the cut-off properties of fundamental mode and second mode by effective area method. Numerical simulation results show that under the same pitch and air filling fraction (AFF), the octagonal photonic crystal fiber has wider single mode operation region and has advantage in designing dispersion-compensating fiber in comparison to hexagonal photonic crystal fiber.