In fiber-based quantum key distribution systems, the photons polarization fluctuation can lead to a decrease of the key generation rate. A method for polarization controlling, based on the genetic algorithm and the numerical control polarization controller is introduced. It is shown that the system could be quickly optimized and the polarization of signal beam has been locked in long term at the single-photons level. In our phase-modulated quantum key distribution system, the average photon number is less than 0.1 photon per pulse. After transmitted through 25 km single mode optical fiber, the polarization state of signal beam is realized to optimum value and keep the stable transmission for a long time, and the final key rate is up to 1.5 kp/s.