For the pixel gradient direction of the top of the head having a fixed scope, this paper firstly selects the candidate pixel points in the foreground. Then it locates the areas of human head-shoulder quickly by these points, which is defined as the windows to be tested. Secondly the illumination invariant color features and the rotation invariant LBP texture feature are extracted and combined together with the Background-weighted Histogram (BWH) algorithm. Lastly, Histogram Intersection Kernel Support Vector Machine (HIKSVM) classifies objects. Experimental results show that based on the pixel gradient direction of the top of the head, the windows which contain head-shoulder can be located more quickly than the traditional method, sliding window, which improves the efficiency of the detection. Furthermore, the accuracy of detection is also improved by the fusion feature of HSV and LBP. Experimental results show that the proposed algorithm is robust and accurate against cluttered dynamical background, occlusion and the object deformation, and tested in many pedestrian datasets and achieved good results.