An efficient and fast convolution neural network for millimeter-wave images that uses deconvolution and a shortcut connection is proposed. The proposed network retains the low-order fine-grained features of the image and significantly improves the detection speed to 27 frame/s from 9 frame/s of original frame. The RCNN (Regions with Convolutional Neural Networks) part of the Faster RCNN is removed. To achieve better network convergence, the initial candidate box size is designed based on thought clustering. The online hard example mining is applied to optimize the loss function of the Faster RCNN such that the imbalance problem between positive and negative samples in millimeter wave images is solved and the training speed is improved significantly. By using the proposed algorithm, the accuracy of 87.6% and the detection rate of 81.2% are obtained on the test set. Compared with mainstream algorithms, the proposed algorithm improves the F1 score by approximately 5%.