The detection and recognition of persons and vehicles in the nighttime environment is highly important in the fields of self-driving cars and security. This paper proposes to use images taken by a cost-effective low-resolution infrared thermal imaging camera. We optimize the faster region-based convolutional neural network according to the unique nature of the images. A multi-channel convolution layer is added to accommodate the grayscale characteristics of thermographic images. We use a global average pooling layer so that fewer images and categories are needed, and we add batch normalization layers to prevent the appearance of exploding or vanishing gradients after the network is widened. The network is trained and tested using 2000 low-resolution thermal images collected in an urban nighttime environment. The average accurate recognition rate is 71.3%, indicating that the method effectively solves the problem of detection and recognition of persons and vehicles in the nighttime environment. The stickiness value and application potential are high.