To handle the problems of illumination change, mutual occlusion of objects, and complicated semantic categories in indoor scenes, a color-depth (RGB-D) image semantic segmentation method based on the dual-stream weighted Gabor convolutional network fusion is proposed in this work. In order to obtain direction and scale invariant features, a weighted Gabor direction filter is designed to construct a deep convolution network (DCN) to extract feature information that is adaptive to direction and scale changes. In order to build a lightweight feature extraction network, a wide residual weighted Gabor convolutional network module is used to extract color and depth dual-stream image features, and a pyramid pooling module is used to fuse the extracted depth features to enrich the image context information. The proposed semantic segmentation method is tested on NYUDv2 dataset, and different comparison methods are set up. The results show that the proposed method is reasonable and effective, and the segmentation effect is competitive.