To ensure detection speed and further improve object detection accuracy, a new model RF-YOLOv2 is proposed on the basis of the YOLOv2 model. In this new model, the KITTI data set is first clustered to select the most suitable number and size of candidate boxes. Next, a residual block structure is used to increase the number of convolutional layers in the training part of the network structure. This increase helps the model to extract more strong features to better describe objects. Finally, a feature pyramid network is introduced in the detection part of the network structure, fusing the feature graphs with different sizes. This network allows even low-level feature graphs to capture rich semantic information. Experimental results show that the RF-YOLOv2 model can gain the deeper information about features and can integrate more object size information. These improvements alleviate significant problems in current models that lead to low detection rates when actual road scenes are complex or when objects vary in shape or structure. The proposed model also improves object detection accuracy in real time detection and achieves better results for large object detection.