Due to the lack of original spectral information, high-resolution remote sensing images are difficult to effectively distinguish various types of vegetation, and the differences between urban and rural vegetation are often ignored and considering that certain vegetation indices somewhat increase the differences among different vegetation types, this paper proposes a deep learning vegetation classification network based on a vegetation index that combines artificial features and spectral information. Based on a parallel network structure, a dense connection module and atrous spatial pyramid pooling module are introduced to enhance the differences in vegetation feature information and effectively improve classification accuracy. Besides, taking full account of the differences between urban and rural vegetation, this paper verifies and analyzes urban and rural areas, respectively. The overall accuracy of urban vegetation classification and extraction is 96.73%, the F1 score is 80.71%, and the intersection-merge ratio is 69.91%. The overall accuracy in classifying and extracting vegetation in rural areas is 91.35%, the F1 score is 90.28%, and the intersection-merge ratio is 82.41%. Each accuracy index exceeds that of other depth learning methods. The results confirm that this method better distinguishes different vegetation types, is suitable for classifying and extracting vegetation from multi-source remote sensing images, and has a definite value for urban green space planning, rural basic farmland supervision, etc.