The effect of transverse mode instability (TMI) is currently the main limitation for the further average-power scaling of fiber laser systems with diffraction-limited beam quality. In this work a main driving force for TMI in fiber amplifiers is identified. Our experiments and simulations illustrate that the performance of fiber laser systems in terms of their diffraction-limited output power can be significantly reduced when the pump or seed radiation exhibit intensity noise. This finding emphasizes the fact that the TMI threshold is not only determined by the active fiber but, rather, by the whole system. In the experiment an artificially applied pump intensity-noise of 2.9% led to a reduction of the TMI threshold of 63%, whereas a similar seed intensity-noise decreased it by just 13%. Thus, even though both noise sources have an impact on the TMI threshold, the pump intensity-noise can be considered as the main driver for TMI in saturated fiber amplifiers. Additionally, the work unveils that the physical origin of this behavior is linked to the noise transfer function in saturated fiber amplifiers. With the gained knowledge and the experimental and theoretical results, it can be concluded that a suppression of pump-noise frequencies below 20 kHz could strongly increase the TMI threshold in high-power fiber laser systems.