To recognize a monitored area automatically for a high-speed railway perimeter-intrusion detecting system, an adaptive image segmentation and recognition algorithm is proposed. The maximum linear feature of each scene is calculated to regulate the adaptive parameters. Moreover, a new combination rule based on the weight of the boundary point and the area size is proposed to rapidly combine the fragmented regions into local areas. A simplified convolutional neural network is designed, the convolutional kernels are pre-trained, and a sparse element is added into the loss function to enhance the diversity of the feature maps. Experimental comparison results indicate that without the graphics processing unit, the pixel accuracy of the proposed algorithm is highest (95.9%), the calculation time is the least (2.5 s), and the number of network parameters is about 0.18×10 6. The proposed algorithm considers an effective balance among the segmentation precision, recognition accuracy, calculation time, manual workload, and hardware cost of the system. Therefore, the automation and efficiency of the railway perimeter intrusion detection system are enhanced.