Laser has many advantages in good monochromatic, high brightness, strong directional and good coherence. Therefore, the integral of laser spectrum can be applied to the field of micro displacement measurement, which is based on the interference principle. In the process of gravimetry, gravity differences caused by different geological structures make the micro nano-scale displacements of detection mass. Therefore, development of micro displacement measurement system with nanometer precision is vital. However, traditional capacitance displacement measurement method (CDMM) is not sufficient to prevent electromagnetic interference. In comparison, the optical interference method (OIM) has the advantages of anti-electromagnetic interference, strong environmental adaptability, and higher precision than CDMM. For the traditional OIM system, the optical path is complex and difficult to integrate, which is unfavorable to the miniaturization and integration of the gravimeter. Therefore, developing a compact OIM system to measure the micro displacement with nanometer precision has become an urgent requirement. A laser interferometric method based on variable phase retardation was designed to achieve sub-nanometer resolution displacement measurement, which has the advantage of compact structure and easy integration compared to traditional OIM system. This system was composed with diode laser, polarizer, analyzer, birefringent crystal group and spectrometer. This paper studies the following aspects: firstly, the measurement system scheme is determined. The structure with dual optical path of polarized light interference is introduced, and the wedge birefringent crystal group is used as the core device, which transforms relative displacement among crystals into differential phase delay between ordinary light (o) and extraordinary light (e). Integrate the laser spectrum, and then the displacement change is transformed to the variance of the synthesis light intensity. Secondly, the physical model of displacement measurement is set. According to the design of birefringent crystal group geometry structure, displacement process and optical path, the relationship between light intensity variance and measured displacement is determined. The third is the optimization of system parameters. In order to make the system measurement error (ME) and measurement range (MR) to meet the practical requirements, using the established physical model, the ME and the MR are set up as a function of crystal cutting angle α and laser wavelength λ. According to application requirements, appropriate bounds of ME and MR is determined, and then α and λ is obtained. Finally, crystal process, systems build and the system measurement test were carried out. In detail, α and λ are chosen as control parameters to optimize and simulate the system, jointly considering “approximate linearization” and “laser intensity fluctuation error”. Meanwhile, jointly considering “laser wavelength fluctuation error” and “laser intensity fluctuation error”, and using the maximum displacement related to system MR, to optimize system ME. Eventually, yαis selected as 20 ° and λ is 635 nm. In testing experiments, displacement measurement is carried out by means of piezoelectric ceramic actuator generating micro-displacement with 10 nm intervals, which includes linear calibration, MR and ME of the system. In addition, a two-hour continuous measurement is carried out using this system when measured position is fixed, and the displacement detection limit is determined by Allan variance. Experimental results show that the displacement MR is longer than 150 nm, and displacement ME is around 0.5 nm, and a detection limit is 0.32 nm @ 23 s, and the linearity determination coefficient R2 is 0.999 85. In conclusion, the system using self-made birefringent crystal group as the core device with adjustable phase retardation can be used as the displacement measurement unit of mass in gravity detection. This system has the advantage of strong environmental adaptability compared with CDMM, and laconic structure and compact light path compared with the conventional laser interference system, so as to facilitate the miniaturization and integration of the gravimeter.