Terahertz (THz) wave is suitable for human body security imaging because of the unique characteristics of THz wave. But the signal noise ratio, constrast and resolution of original terahertz images need to be improved presently. In order to improve the quality of the THz images for security screening, a passive THz imaging restoration algorithm based on Markov random field is put forward. After being pre-processed by image de-noising and enhancement treatments, the original image is restored based on Bayes analysis to which Markov constraint is added. Different definition of resulting images are obtained by changing iteration and regularization parameter. Optimum parameters are determined by objective standard evaluation and subjective visual effect. Experimental results show that the proposed method achieves good balance between noise removing and edge information preserving of passive terahertz image. Thus target resolving ability of terahertz security images is improved dramatically.