The flame pattern is artificially designed by most of the traditional flame detection methods based on physical signal of the flame, and is identified according to the pattern recognition method. These methods are easy to be interfered by the external environment. Because the generalization of artificially designed flame feature is not strong, the recognition accuracy will reduce when the flame shape or scene changes violently. To solve this problem, a method of deep learning for detecting the flame based on the regional full convolution network (R-FCN) with the residual network (ResNet) is proposed. The feature is extracted automatically by the feature extraction network, and the flame position is determined by R-FCN, and it is secondary classified by ResNet for further reducing the false alarm rate. The proposed method, which eliminates the feature extraction process of the traditional flame, realizes end-to-end automatic acquisition of flame characteristics and performs corresponding detection processes. An average recognition accuracy reaches to 98.25% in the flame video data set of Bilkent University.