The electronic structures and optical properties of transition metal (TM, TM refers to Fe, Co, and Ni, respectively) doped GaSb are studied by the LDA+U method of the first-principles calculation. The results indicate that these TMs can enhance the absorption amplitudes of GaSb semiconductors in the infrared region, and improve the photocatalytic performances of GaSbs effectively. For the doped systems, TMs tend to substitute for Ga and form TM@Ga defect. The charge layout and bond population of TMs imply that the electric dipole moment induced by lattice distortion separates photoelectrons from holes to some degree, and consequently enhancing the photocatalytic performance. The impurity levels induced by TMs are close to the Fermi level, which illustrates that the imaginary part of complex dielectric function has the capability of response when the energy of photon is zero. Meanwhile, the static dielectric constant of the doped system is also enhanced compared with that of the un-doped system. The doped TMs can improve the optical properties of GaSb systems for three dopants effectively, but the Ni dopant is the best for the photocatalysis properties of GaSb in the three dopants. The further analysis shows that the uniform Ni can hinder the recombination of electron-hole pairs, and the optical absorption range and absorption peak are both biggest when Ni molar concentration is 10.94%, which is favorable for photocatalytic performance. Our results will extend the applications of GaSb to the fields of infrared thermal photovoltaic cells, infrared light detector, and infrared semiconductor laser.