Aiming at the problem that the traditional convolutional neural network (CNN) algorithms only recognize semantics of one-hand gestures and the problems of the poor convergence and low recognition accuracy of the deep learning gesture recognition algorithm, an adaptive one-hand and two-hand gesture recognition algorithm based on double classifiers is proposed to recognize single-hand and two-hand gestures. The core of the algorithm is combining two classifiers for single-hand and two-hand gesture recognition. First, the hand number classifier is used to segment and group the gestures, and the gesture recognition is converted into partial gesture image recognition. Second, the adaptive enhanced convolutional neural network (AE-CNN) is used for gesture recognition, and the adaptive module analyzes the cause of the recognition error and feedback mode. Finally, the parameters are updated based on the number of iterations and recognition results. Experimental results show that the correct probability of the hand number classifier for gesture prediction grouping is 98.82%, the convergence of AE-CNN is better than that of CNN and CNN+Dropout, and the recognition rate of one-hand gestures is as high as 97.87%. The overall model recognition rate of 9 types of single-hand gestures and 10 types of double-hand gestures built based on LSP dataset is 97.10%, and the average recognition rate of gestures under complex backgrounds and different light intensities is 94.00%. The proposed algorithm has certain robustness.