Fringe projection profilometry (FPP) is widely used in optical three-dimensional (3D) measurements because of its high stability. In FPP, fringe distortion is an inevitable and highly complex systematic error that significantly reduces the 3D measurement accuracy. At this point, the existing causes of fringe distortion represented by gamma distortion, high-order harmonics, and image saturation have been effectively analyzed and compensated to restore high-quality fringe images. In this paper, we innovatively reveal a concealed cause of fringe distortion, i.e., intensity diffusion across pixels, which is induced by photocarrier diffusion between photodiodes. To the best of our knowledge, intensity diffusion has not been studied in the field of fringe restoration. Based on the motion of photocarrier diffusion, we theoretically analyze the mechanism of how the intensity diffusion affects FPP. Subsequently, an intensity diffusion model is established for quantifying the diffused intensity in each pixel, and an intensity diffusion correction algorithm is presented to remove the diffused intensity from the fringe images and correct the fringe distortion. Experiments demonstrate the impact of intensity diffusion on FPP, and the 3D measurement results prove the effectiveness of the proposed methods on improving the 3D measurement accuracy by correcting the fringe distortion.