To improve the accuracy of the automatic detection and classification of apples on the tree, image features and artificial neural network classifier are applied to segment the apple images. First, apple image samples and background image samples are chosen. Then the color feature and the texture features of the samples are calculated. The color feature (R/B ratio) is calculated based on RGB color model, and the texture features (contrast and correlation) are calculated by gray level co-occurrence matrix (GLCM). These three parameters are used as the input to the back-propagation neural network (BPNN) classifier. The result of the output layer is a numerical value in the runge of 0~1. It is classified into fruit and background based on a certain threshold value. The results of the segmentation show that the success rate is over 87.6%, and the influence of light is neglectable. It is feasible to use the algorithm in practical recognition of apple.