The holographic retinal projection display (RPD) technology based on wavefront modulation can converge an image beam without a lens and realize the free adjustment of the image depth of field; position, number, and spacing of viewpoints; and other parameters. At present, the amplitude-type hologram is mostly used to explore the lensless wavefront modulation hologram RPD; however, using this type of hologram involves the problems of low diffraction efficiency and conjugate image crosstalk. Hence, a calculation method of the phase-only hologram of the lensless holographic RPD is proposed. The required hologram can be obtained in two steps. First, the target image is multiplied by the uniform phase as the input. Upon performing the Gerchberg-Saxton iteration based on angular spectrum diffraction between the input surface and the hologram surface, the phase distribution is obtained on the hologram surface. Further, the phase distribution after iteration is multiplied by the phase of the converged spherical wave and encoded to obtain the final hologram. In the experiment, the color RPD was realized by the time division multiplexing technology, and the exit pupil expansion was realized by the multi spherical wave phase; thus, the effectiveness of the proposed method was verified.