To enhance the convergence ratio, timeliness and accuracy of current iterative algorithm in the measurement of three dimensional absolute displacements with point-diffraction interferometer (PDI), the method based on quick searching particle swarm optimization (QS-PSO) algorithm is proposed for the reconstruction of absolute displacements. The three dimensional absolute displacement of the probe is determined from the distribution of the phase difference in the interference field. To realize the efficient procession of a large number of sampling pixel points, the sampling points are nonlinearly increased in the process of iteration in the proposed QS- PSO method, by which both the high- efficiency and high- precision measurement are achieved. Both the computer simulation and experimental measurements are carried out to verify the feasibility of the measurement. According to the analyzing results, the convergence rate and measurement precision of the proposed measurement method can reach 90% and better than the order of microns within a 200 mm×200 mm×300 mm working volume, respectively. The proposed absolute displacement measurement method is of high efficiency, measurement precision, anti-noise ability and reliability, and also of great significance for the application of micro-processing and high precision measurement.