Existing deep learning-based terrain classification methods are mainly for remote sensing imagery； however， the spatial information of point clouds is underutilized. Specifically， the fusion of heterologous features is insufficient for point clouds and imagery. To utilize multi-source features fully， we propose a self-adaptive fusion classification method of multi-source remote sensing data based on independent branch network in this study. First， three-dimensional （3D） and two-dimensional （2D） networks are used to extract the semantic features of registered LiDAR point clouds and remote sensing imagery. From the 3D space， the features of imagery are then sampled and aligned with those of point clouds. Finally， a nonlinear self-adaptive feature fusion module is proposed to realize the fusion of multi-source semantic features. The experimental results indicate that the proposed method achieves an average classification accuracy of 85.87% on the vegetation， building， and ground of the ISPRS multi-source remote sensing dataset. Through network training， multi-source remote sensing data can be more data feature-adaptive fused and classified； further， the accuracy is significantly improved by 10.12% compared with the 3D classification result. The proposed independent branch fusion network can realize interactive learning and deep fusion of 2D and 3D data， and it provide a new idea for terrain classification based on remote sensing multimodal data fusion.