A distortion correction method for the elemental images of integral imaging (II) by utilizing the directional diffuser is demonstrated. In the traditional II, the distortion originating from lens aberration wraps elemental images and degrades the image quality severely. According to the theoretical analysis and experiments, it can be proved that the farther the three-dimensional image is displayed from the lens array, the more serious the distortion is. To analyze the process of eliminating lens distortion, one lens and its corresponding elemental image are separated from the traditional II. By introducing the directional diffuser, the aperture stop of the separated optical system changes from the eye’s pupil to the lens. In terms of contrast experiments, the distortion of the improved display system is corrected effectively. In the experiment, when the distance between the reconstructed image and lens array is equal to 120 mm, the largest lens distortion is decreased from 46.6% to 3.3%.