Fusion super-resolution reconstruction methods of red-green-blue (RGB) images and hyperspectral images have shortcomings such as inadequate utilization of the pixel structural and spectral similarities of images, intrinsic space loss during image scaling, and loss of spectral information and contextual relationships. To address these issues, we propose the SparseVAFormer graph convolutional model. We first utilize composite graph convolution to capture local detail features by leveraging the spatial relationships of pixels and then modeled the correlation between different spectra, thereby enabling full exploration of the high-dimensional characteristics and non-Euclidean structural information of images. We then construct the VAFormer module to map the data to a low-dimensional latent space to capture the core features of images. Through self-attention mechanisms, the model considers all pixels in the entire image when computing the representation of each pixel, thereby capturing complex long-distance spatial and spectral dependency relationships between pixels. This process enables the model to simulate the spectral reflection characteristics of real hyperspectral images. Finally, we design a multi-scale mixed convolution module to strengthen the flow of differential information between different levels and channels, thereby assisting the model in capturing complex features ranging from subtle textures to large-scale structures. Experimental results demonstrate that the proposed model achieves the best peak signal-to-noise ratio of 51.299 dB and 49.762 dB on the CAVE and Harvard datasets, respectively. Thus, the sparse VAFormer graph convolutional model can effectively fuse multi-spectral and hyperspectral images, outperforming some advanced models in the field of hyperspectral image super-resolution such as FF-former and LGAR.