This paper proposes a method for the scene classification of optical remote sensing images based on the residual network of convolutional neural networks. In the proposed method, two modules, i.e., jump connection and covariance pooling, are embedded in the original network model to achieve multiresolution feature mapping and combine different levels of multiresolution feature information. Experiments are conducted on three open classical remote sensing datasets. Results show that the proposed method can fuse the multiresolution feature information of different levels in the residual network and use higher-order information to achieve more representative feature learning. The proposed method exhibits higher classification accuracy in the scene classification problem compared with the existing classification methods.