High-risk work site environments are complex and dangerous and responsible for many fall accidents and casualties. To detect the fall behavior of workers, a human fall detection method using a Kinect sensor was proposed. Based on depth images obtained using a Kinect, we extracted body joint points information and determined whether a human body fell by calculating the changes of the relative position entropy and speed of the joint points. Through comparative experiments, a set of skeleton joint points with the highest fall recognition rate were determined: head, shoulders, knees, and center points. Experimental data show that the method can detect fall behaviors more quickly and accurately compared with the conventional methods.