In order to realize the ultratrace detection of Hg2+ by optical waveguide spectroscopy, a displacement strategy of gold nanoparticles (Au NPs) for target analysis based on the optical waveguide spectroscopy is designed, which differs from the usual target-induced aggregation effect. The PATP-coupled Au nanoparticle (PATP-Au) multilayer was layer-by-layer assembled on the OWG chip surface and used as the probes to detect Hg2+. OWG spectroscopy can monitor the SPR extinction from Au nanoparticles, which enables to easily capture changes in the probe structure. Because Hg2+ has a higher binding affinity to the thiol group of PATP molecules, the presence of Hg2+ will greatly destroy the probe structure, produce large changes in surface plasmon resonance features of the probes. Our experiments indicated a high selectivity for Hg2+ detection and reached a low detection limit of about 1 nM.