Quickly and efficiently distinguishing and eliminating weeds is one of the keys to improving the extraction accuracy of pitaya plants. In this study, a high-resolution aerial image is acquired using a four-rotor unmanned aerial vehicle (UAV) platform with a visible light lens. The spectral characteristics of pitaya plants and weeds in R, G and B channels are then analyzed, and the close color difference vegetation index (CCVI) is constructed based on the pixel digital number (DN) values. Through OTSU threshold segmentation, majority/minority analysis and cluster hole filling, the mainstream indices including VDVI, EXG, and NGRDI are compared with CCVI. Results show the following: 1) for the pitaya plant plot with a high or full coverage rate for weed, the CCVI extraction effect is better, whereas the other three indices have poor classification effect in the plot where weeds and plants coexist; 2) for the three research ROI samples, the overall average accuracy and the Kappa coefficient are 94.60%, 0.9417, respectively, and for the test sample, the overall extraction accuracy and the Kappa coefficient are 94.33% and 0.9328, respectively. Thus, it is verified that the extraction accuracy of plants with similar habitats in different regions is fairly similar. Results confirm that the CCVI can be used to identify and extract the individual pitaya plants from the weeds with the UAV remote sensing scheme, and its extraction effect is good. The proposed method can be applied in conjunction with VDVI, EXG, and NGRDI.