For the applications of efficient high-precision object detection in optical remote sensing (RS) images, this paper focuses on the difficulty of improving the detection accuracy of the SSD (single shot multibox detector) model on small and densely distributed objects in such images. An improved model FFC-SSD (multi-scale feature fusion & clustering SSD) is thereby proposed. For this purpose, a bounding-box group clustering (BGC) module is designed. Group clustering is implemented to obtain default object frame parameters that are more consistent with the size distribution of object samples and gives more attention to small objects. This module effectively improves the network’s ability to extract object locations. Then, an efficient de-pooling multi-scale feature fusion (MSFF) module is designed to enhance the ability of the model to extract object features and effectively reduce the efficiency loss of the model at the same time. The experimental results demonstrate the effectiveness and applicability of the FFC-SSD model for object detection in optical remote sensing images. The proposed model achieves a favorable balance between precision and efficiency and has high detection accuracy on small objects.