The interpretation of single remotely sensed data source may suffer from inaccurate boundary and low classification accuracy. The integration of hyperspectral and LiDAR data opens up the possibility to improve the classification performance. But, it is a challenge that how to appropriately integrate the considerable heterogeneity between the two types of data. In this paper, a conditional random field classification method was proposed to solve this problem by jointly taking both the heterogeneity of fused spectral-spatial-height features and co-occurrence of class labels into account. Firstly, the morphological features were extracted from two types of data respectively, and a graph model and training samples were jointly used to fuse the morphological features and spectral features. The obtained features were inputted into a support vector machine classifier to obtain the initial classification results with probabilistic outputs. Then, based on the fusion features, a local heterogeneity value was calculated to measure the essential difference of classes among pixels. Meanwhile, a class co-occurrence matrix, whose element calculated the spatial relationship between classes, was also obtained. Finally, a conditional random field framework was used to integrate the initial classification results, local heterogeneity information and the class co-occurrence matrix, and obtain the final classification results through inferencing two objective functions. In this process, by defining the weight between two neighboring pixel as a monotone decreasing function respect to the normalized Euclidean distance of the corresponding fused features, the object boundary could be regularized by giving a smaller weight to the class pairs with different labels and distinct features. Similarly, by giving a small weight to the class pairs with a strong spatial relationship, the purpose of maintaining the class pairs with stable spatial relations could be achieved. The method was validated with Houston and Gaofeng forest farm data sets. The overall accuracies of the proposed method reached to 94.00% and 92.84% respectively, and the "pepper and salt" phenomena of the initial classification results were significantly reduced. The result indicates the effectiveness of the proposed method.