Full-chip multi-parameter optimization is an important development direction of resolution enhancement techniques in optical lithography. A source mask projector optimization (SMPO) method based on particle swarm optimization (PSO) algorithm is proposed. The pixels are used to represent source. The discrete cosine transform basis functions are used to represent the mask. The coefficients of Zernike polynomials are used to represent the projector. The source, the mask and the projector are encoded into particles. The pattern error is adopted as the evaluation function and the particles are updated iteratively to realize the SMPO. This method is simulated and verified by using the complex mask pattern with cross gate design in nominal condition and process condition. Results show that the pattern errors are reduced by 94.2% and 93.8%, respectively, and the quality of lithography imaging is effectively improved. Compared with SMPO method based on genetic algorithm, the proposed method has a faster convergence rate. Besides, the proposed method has the advantages of high degree of optimized freedom and enhanced manufacturability of the optimized mask pattern.