A method for enhancement of a single shot multibox detector (SSD) for aerial infrared target detection is proposed. Herein, the relationship between the sensing field and number of feature layers is analyzed, and a bidirectional feature map fusion mechanism that uses both pooling and deconvolution operations is proposed to enhance the feature expression ability. The semantic enhancement branch of the shallow feature map is introduced and the prediction boxes on the high-resolution feature map are increased, so that the positing accuracy of small-size targets is improved. Comparative experiments on the VOC2007 small object dataset and an aerial infrared target dataset reveal that the mean average precisions increase by 7.1% and 8.7%, respectively, accompanied by a slight decrease in detection speed. The results demonstrate that SSD enhancements can achieve good performance in aerial infrared target detection.