A novel image inpainting forensics algorithm based on the deep neural network is proposed, in which the vestigial features can be automatically extracted by the encoder network, the category of each pixel is predicted by the decoder network, and thus whether or not the image is with inpainting and falsification as well as the inpainted and falsified regions can be distinguished. Simultaneously, the feature pyramid network (FPN) is used to supplement the feature map in the decoder network. The MIT Place dataset is used as the training set and the UCID dataset as the testing set. In addition, the different inpainting and falsification algorithms are adopted for the training set and the testing set, respectively. The experimental results show that, compared with the other inpainting forensics algorithms of images, the proposed algorithm has a more accurate inpainting area and a faster processing speed. Moreover, it has relatively good robustness and strong generalization ability against different inpainting forensics algorithms.