Cavity optomechanics is applied to study the coupling behavior of interacting molecules in surface plasmon systems driven by two-color laser beams. Different from the traditional force–distance measurement, due to a resonant frequency shift or a peak splitting on the probe spectrum, we have proposed a convenient method to measure the van der Waals force strength and interaction energy via nonlinear spectroscopy. The minimum force value can reach approximately 10?15 N, which is 3 to 4 orders of magnitude smaller than the widely applied atomic force microscope (AFM). It is also shown that two adjacent molecules with similar chemical structures and nearly equal vibrational frequencies can be easily distinguished by the splitting of the transparency peak. Based on this coupled optomechanical system, we also conceptually design a tunable optical switch by van der Waals interaction. Our results will provide new approaches for understanding the complex and dynamic interactions in molecule–plasmon systems.