The different fluorine impurities in phosphogypsum have varying effects on the microstructure and properties of hardened gypsum paste, and their mechanisms are not yet clear, which affects the recycling of phosphogypsum resources. In this paper, through testing methods such as setting time, in-situ hydration heat, ion concentration, mechanical properties, mercury intrusion, X-ray diffraction, X-ray photoelectron spectroscopy, and scanning electron microscopy, the effects of four types of fluorine impurities (CaF2, NaF, Na2SiF6, and Na3AlF6) on the hydration process, microstructure, and mechanical properties of calcined gypsum were systematically studied. The results show that, soluble fluorine impurities can promote the hydration of calcined gypsum, showing a certain effect of promoting coagulation. The higher the solubility of fluorine impurities is, the more significant the promotion effect on the hydration process of calcined gypsum (NaF>Na3AlF6>Na2SiF6) is. Insoluble CaF2 has little effect on the hydration process of calcined gypsum. However, excessively fast hydration rate can lead to premature hardening of the slurry, preventing some calcined gypsum from hydrating promptly. Consequently, these calcined gypsum may gradually transform into plate-like crystals during subsequent slow hydration, increasing the porosity of the hardened gypsum paste and impairing its mechanical properties. This study provides certain guidance for the efficient utilization of phosphogypsum in building materials.