The characters of different Yb3+-doped fiber materials are integratedly studied and the single-frequency power limits of Yb3+-doped silica fiber, phosphate fiber, YAG crystal (or ceramic) fiber and sapphire fiber are carefully analyzed. The numerical result shows that the extractable power of YAG crystal (or ceramic) fiber and sapphire fiber can be more than 10 kW. The further study of the power limit shows that YAG crystal (or ceramic) fiber and sapphire fiber also have an advantage in single-mode working condition, indicating that these two kinds of fibers still have potentials to power scaling. For the fiber material of single-frequency fiber laser, the effective ways to boost the power limit are to reduce the stimulated Brillouin scattering (SBS) gain coefficient and improve the performance of heat conducting.