The observation and recognition of sunspots is an important task of solar physics. By observing and analyzing sunspots, solar physicists are able to analyze and predict solar activities with higher accuracy. With the continuous progress of observation instruments, solar full-disk image data amount is also on a rapid growth. In order to recognize and label sunspots quickly and accurately, a two-layer sunspot recognition model is proposed in this paper. The first layer model is based on deep learning model YOLO. In order to enhance the ability of YOLO to recognize small sunspots, the parameters of YOLO are optimized by using the k-means algorithm based on inter-section-over-union. The final YOLO model can identify most large sunspots and sunspot groups, with only a few isolated small sunspots being unidentified. For the purpose of further improving recognition rate of small sunspots, the second layer model applies AGAST (adaptive and generic accelerated segment test) feature detection algorithm to specifically identify the missing small sunspots. The experimental results on SDO/HMI sunspot data set show that all kinds of sunspots can be recognized effectively with high recognition accuracy by using the model proposed in this paper, thus realizing the real-time sunspot detection task.