We demonstrate the optomechanical cooling of a tapered optical nanofiber by coupling the polarization of light to the mechanical angular momentum of the system. The coupling is enabled by birefringence in the fiber and does not make use of an optical resonator. We find evidence for cooling in the distribution of thermally driven amplitude fluctuations and the noise spectrum of the torsional modes. Our proof-of-principle demonstration shows cavity-less cooling of the torsional degree of freedom of a macroscopically extended nanofiber.