In the field of solar power generation such as solar water heaters and solar cells, the climate factors such as rain, snow and cloudy days will seriously affect the power generation effect, and the work of solar servo system must also consume the energy. Therefore, it is extremely important to quickly judge the current weather conditions and design an adaptive on-off servo system. When the weather is rainy or snowy, the system should be shut down to reduce the energy consumption. In view of the problems of low efficiency, poor accuracy and large amount of calculation of traditional weather recognition methods, a weather classification set with multiple categories on the basis of public weather images was created, and a weather image recognition technology based on convolutional neural network and feature fusion was provided. By using the traditional way to obtain the color, texture and shape of the image as the bottom features of the whole model, it was improved on the basis of the original visual geometry group-16 (VGG16) model, so as to extract the deep features of the image. Finally, the bottom features and deep features were fused and output on Softmax, and the total recognition rate is 94%.