To optimize the effect of conventional beam shaping algorithm for designing Phase Diffractive Optical Elements (PDOE), a new hybrid algorithm is presented. It combines the Genetic Algorithm (GA) with transformable scale BFGS (Broyden, Fletcher, Goldfarb, Shanno) algorithm, and penalty function is also used in the cost function conformation. The hybrid algorithm has the global nature of the genetic algorithm as well as the local improvement capabilities of the BFGS algorithm. Compared with other beam shaping algorithms, it has good efficiency, good convergence property and good stability. As a design example, both the conventional simulated annealing algorithm and hybrid algorithm are used into Gaussian beam shaping. To review the reliability of two algorithms, two indexes of diffractive efficiency and Signal-to-noise Ratio (SNR) are considered in numerical simulation. The design results show that hybrid algorithm has good convergence speed and cost function. It can acquire good diffractive efficiency and SNR with only a small quantity of iterative numbers. The shaping effect of diffractive phase optical elements designed by the hybrid algorithm is greatly improved. It will be widely used when the high uniformity is needed.