In this work, we analyze the effects of the background spectral noise in phase-modulated single-frequency seed lasers on the spectral purity of high-power narrow-linewidth fiber amplifiers. Through demonstrating the spectral evolution of the phase-modulated single-frequency part and the background spectral noise in a narrow-linewidth fiber amplifier, the mechanism for the spectral wing broadening effect is clarified and design strategies to maintain high spectral purity are given. Specifically, the background spectral noise in phase-modulated single-frequency seed lasers could lead to obvious spectral wing broadening and degeneration of spectral purity in narrow-linewidth fiber amplifiers through the four-wave-mixing effect. Notably, the spectral wing broadening effect could be suppressed by filtering out the background spectral noise in the seed laser or applying a counter-pumped configuration in the fiber amplifier. We have also conducted contrast experiments, which have verified the validity of the theoretical model and the design strategies for high-spectral-purity operation.