The method on switching the direction of spin accumulation of spin Hall effect of light in reflection is studied theoretically and experimentally. A model describing spin accumulation of reflected light at the interface between air and prism is established. And the qualitative relation between traverse shift and the polarization angle of incident beam is revealed. The direction of spin accumulation will reverse with the increase of the polarization angle of incident light on condition that the incident angle is fixed and less than Brewster angle. But the direction of spin accumulation won′t flip with polarization angle if the incident angle is bigger than Brewster angle. The results show that the direction of spin accumulation can be switched by adjusting the polarization angle of incident light while the incident angle is fixed and less than Brewster angle. The study about the direction of spin accumulation provides a new pathway for modulating the spin Hall effect of light.