We study the power limit of holmium-doped fiber lasers with the consideration of thermal effects, optical damage effect, brightness of pump source and nonlinear effects. Power limits and physical limits of holmium-doped silica fiber lasers in both pump regions (pumped with ytterbium-doped or thulium-doped laser source) are calculated. It is found that using both of the two pump regions, separately, based on current technical conditions, power limits of broad bandwidth holmium-doped fiber lasers are 29.9 kW and 70.2 kW, respectively. Limits to power now are, in fact, brightness of the pump sources, thermal lens effect and stimulated Raman scattering (SRS), and thus to improve the brightness of the pump sources is an effective method to promote high output power of holmium-doped fiber lasers. The situations of fiber lasers with single-frequency and strict single-mode operation are considered. At the same time, fiber laser pumped with a thulium-doped laser source has an advantage for achieving higher output power.