Pedestrian detection is one of the most important tasks of robots and unmanned vehicles at nighttime. Faster region convolution neural network framework is used to realize the pedestrian detection of infrared image at nighttime. This framework uses region proposal network to generate region proposals. Therefore, it is unnecessary to generate region proposals separately from the image. The parameter sharing mechanism is adopted in the convolutional layers in region proposal network and convolutional network for classification and bounding box regression, which makes the framework an end-to-end advantage. Thus, the pedestrian detection can be implemented from the input image to the detection result directly and it is unnecessary to manually select the features of the target. Experimental results show that the proposed method increases the recognition accuracy from 68.2% and 73.4% to 90.9% and shortens the recognition time from 3.6 s/frame and 2.3 s/frame to 0.04 s/frame compared with the traditional method and fast region convolution neural network, respectively, which reaches the required real-time level in practical applications.