In order to avoid the error introduced by the imaging system in traditional pinhole type point diffraction interferometer, a lensless imaging algorithm is proposed. The algorithm can retrieve the phase of the interferograms captured by CCD without the imaging system. On the basis of the plane wave spectrum theory, we can obtain the interferogram of the exit pupil conjugate location by constructing a virtual imaging conjugate relation. Simulation and experimental results show that the proposed lensless imaging algorithm can remove the edge diffraction fringes and the central diffraction effects in the interferogram, and the coherent noise can be eliminated as well. With this lensless imaging method, we can remove the interference imaging system in the pinhole type point diffraction interferometer which is difficult to design and fabricate. Besides, this algorithm can also effectively guarantee the high accuracy of pinhole type point diffraction interferometer.