Compared with the previous two types of single-frame image super-resolution reconstruction algorithm, the super-resolution with convolution neural network (SRCNN) has greatly improved the operational efficiency and recovery accuracy with its end-to-end mapping structure. However, the number of hidden layers and the convergence performance of the network make the recovery effects of some images worse than the example-based reconstruction algorithms. In view of the problem of network optimization, the algorithm of combining particle swarm optimization (PSO) with SRCNN is proposed. PSO is used to initialize the network weight and the gradient descent (GD) algorithm is used to correct the weight which can combine the global search capability of PSO and the local search ability of GD. The experimental results of set5, set14 datasets and the blurred images under haze weather respectively show that the proposed algorithm can not only use less parameters to obtain higher performance network, but also has better reconstruction effect than the existing four algorithms, and the ability to sharpen edges is stronger.