Human action recognition in videos has attracted more and more attentions. In view of the local expression of human behavior, a novel local contour feature representing body posture is proposed, which can make full use of information of the contour variation along both horizontal and vertical direction. The proposed local feature can distinguish different actions and is invariant to translation, scaling, rotation and change of start point of human contour. A two stage classifying framework based on random forest is also proposed by using this novel local body contour feature. Random forest is employed to classify each frame of the test video. After that, a video classification method based on out of bag(OOB) error weighted voting strategy to recognize action video according to the ratio of decision trees belonging to each local contour to total decision trees is proposed. Experimental results on test data set prove the effectiveness of proposed method.