Hand tracking is the basis and core problem of vision interaction in virtual reality. Due to the poor performance resulted from the existing hand tracking methods in the instances of movement, scale change, complex background, etc., a new hand tracking algorithm is presented. The proposed algorithm is under particle filtering and tracking framework, which adopts oriented gradient local binary pattern descriptor integrating with texture and contour information. Furthermore, the infrared depth information is introduced. The proposed tracking algorithm combines the observation information of current frame in the stages of particle sampling and updating by the artificial bee colony algorithm, which overcomes the degeneracy problem in particle filtering and improves hand tracking precision by optimization of the space search. Experimental results show that the proposed algorithm can achieve accurate and robust hand tracking in complex background.