Aiming at the color distortions generated by color space conversion and the strong correlation in the red green blue (RGB) space during image fusion process. The fusion framework is proposed based on the improved two directional two dimensional principal component analysis [(2D)2PCA], which overtakes the shortcomings of PCA in catching image structure and reducing spectral information lost. Considering the structure of images in RGB space, the rows and columns of input images are set as the inputs of two 2DPCA approaches. The reconstruction weights of row and column directions are set linearly to the covariance. The PC replacemet is based on the structure properties of the reconstruction. The fusion is built by weighting reverse transformation of covariance. To verify the effectiveness of the proposed method, two experiments are discussed. One experiment uses the high resolution grey image and its responding blurred color image as source images, the other experiment is built on the visual color image and the infrared image. Experimental results show the superior of the proposed method over previous works with respect to the spatial resolution as well as other fusion indicators.