Erbium-doped holey fiber with hexagonal lattice was modeled by using effective index method. In order to calculate the equivalent step index of the periodic structure of the cladding holey optical fiber, all-vectorial fundamental space filling mode approach was utilized. By using EH11 mode, we have numerically solved the rate equations of a three-level pumping scheme for a fiber laser. The obtained results have shown a good agreement with the other experimental results, recently. The results have predicted amplifiers with gain efficiencies as high as 10 dB/mW.