Component substitution is a classical method for remote-sensing image fusion that has good spatial fidelity but is prone to spectral distortion. Therefore, a panchromatic and multispectral image fusion method that combines structural and energy information is proposed. First, the method decomposes the spatial and spectral information of multispectral images by hyperspherical color-space transformation. Second, a two-layer decomposition scheme is introduced through joint bilateral filtering. The panchromatic image and intensity components are then decomposed into structural and energy layers. Finally, the structural layer is fused by the neighborhood spatial frequency strategy, and the pure energy layer of the intensity component is used as the energy layer of the pre-fusion image. The intensity component defines the color intensity. By combining the pre-fused structural layer with the energy layer of the intensity component, the spatial and spectral information of the source image can be effectively combined, thereby reducing the spectral distortion of the pansharpened image. In this study, several experiments were conducted on the Pléiades and QuickBird datasets, and the experimental results were qualitatively and quantitatively analyzed. The results show that the proposed method has certain advantages over existing methods.