In infrared image detection and segmentation tasks, the color information is lost, the features are fuzzy with noise, the target number is large, and the traditional extraction method is slow. Therefore, we propose an optimized YOLO detection and segmentation network model for far-infrared images. The two proposed optimization points are as follows. We use the K-means++ clustering algorithm to determine the multi-scale prediction anchor size after the analysis of two far-infrared databases. We also perform pixel-level instance segmentation of detection targets using localized adaptive threshold segmentation. The experimental results show that the proposed algorithm performs pedestrian detection at detection speeds of 29 frame/s and 28 frame/s on the FLIR dataset and the dataset used in this paper, respectively, ensuring the requirement of real-time output. The pedestrian detection accuracies in these datasets reach 75.3% and 77.6%. Moreover, the average intersection over the union of the segmentation results is 70%--90%. In summary, the algorithm performs well with respect to robustness and universality. The algorithm provides a valuable reference method for pedestrian detection and segmentation in far-infrared fields.