Aiming at the problem that the image quality fluctuation of passive terahertz security system is caused by environmental change, which affects the recognition algorithm and leads to a significant decrease in accuracy, this paper proposes an improved YOLOv4 algorithm based on Focal-Efficient Intersection Over Union(EIOU) loss function, and uses passive terahertz human security image to conduct model training for prohibited items of knife and gun. A terahertz image database of people carrying suspected objects in different environments and different positions is established, and a rich data set is constructed by image augmentation method. The Complete IOU(CIOU) loss of YOLOv4 is improved to Focal-EIOU loss to improve the robustness of the algorithm for terahertz image recognition, and then a better model is obtained after training. In the test set of this paper, since YOLOv4 algorithm has low robustness for terahertz image recognition accuracy, CIOU loss of YOLOv4 is modified and adjusted to Focal-EIOU loss, and a better model is finally obtained through training. The mean Average Precision (mAP) of the model trained by the improved algorithm reaches 96.4%, the detection speed is about 28 ms, and the average value of IOU is 0.95, which are higher than those of the conventional algorithms under the same conditions, the detection and recognition effect are improved. The experimental results show that the proposed method can effectively improve the suspect identification accuracy of passive terahertz human security system, which is conducive to the popularization and application of this technology in the field of human security.