In this paper, a theoretical scheme to realize nonreciprocity in a three-cavity optomechanical system was proposed. Three optical cavity fields propelled by strong driving light fields were individually integrated to a mechanical oscillator through radiation pressure, and two were driven by weak probe light fields when they were coupled with an optical fiber. Through the Heisenberg-Langevin equation, the steady-state solution of the three-cavity optomechanical system was presented. The specific expression of the transmission amplitudes is obtained using the input-output theory. The results reveal that the nonreciprocity in the three-cavity optomechanical system is because of the quantum interference between the optomechanical interaction and the coupling interaction of two optical cavity fields. The phase difference not only determines whether the nonreciprocity can occur in the system but also determines the direction of the nonreciprocity. Furthermore, it is also discovered that with an increase in the effective optical coupling strength, the transmission amplitude curves change in different forms. Under a certain effective optomechanical coupling strength, the system achieves the perfect nonreciprocity. Our research results can provide reference for the application of quantum information processing based on a cavity optomechanical system.