A novel interpolation-based subpixel mapping for hyperspectral image by using pansharpening

Peng WANG; Hong-Yu YAO; Gong ZHANG

doi:10.11972/j.issn.1001-9014.2021.01.010

Abstract

In this paper, a novel interpolation-based subpixel mapping (ISPM) for hyperspectral image by using pansharpening (PAN-ISPM) is proposed. In the proposed method, a novel processing path is added into the existing processing path of ISPM. Firstly, the original coarse hyperspectral image is improved by pansharpening technique in the novel processing path, and the novel fine fraction images are derived by unmixing the improved image. Secondly, the novel fine fraction images from the novel path and the existing fine fraction images from the existing path are integrated to produce the finer fraction images with more spatial-spectral information. Finally, according to the predicted values from the finer fraction images, class labels are allocated into subpixel to obtain the final mapping result. Experimental results show that the proposed method produces the higher mapping accuracy than the existing ISPM methods.

Keywords

hyperspectral image pansharpening technique spatial-spectral information subpixel mapping

Introduction

Due to limitation of hardware and complexity of environment，hyperspectral image always contains lots of mixed pixels，resulting in the inaccurate land cover class mapping information^［1］. Subpixel mapping（SPM）^［2］ can explore the spatial distribution information of the land cover class at subpixel scale. In SPM，a mixed pixel is changed into $S \times S$ subpixels by the zoom factor $S$ ，class labels are assigned to subpixel to achieve the mapping results. SPM transforms the fraction images（spectral unmixing results of hyperspectral image）to a hard classification image^［3］.

There are two main SPM types：initialization then optimization type and soft then hard type^［4］. SPM based on pixel swapping^［5］，perimeter minimization^［6］，value of neighbouring^［7］ and Moran’s index^［8］ all belong to the initialization then optimization type. In this type，the class labels are first randomly assigned to subpixel. The location of subpixel is then gradually transformed by optimization algorithm. Simulating annealing，particle swarm，and genetic algorithm^［9，10］ are selected as optimization algorithm for this type. However，this type often requires a lot of computation time due to its complex physical structure. The more popular type is soft then hard type. Soft then hard type mainly contains back propagation neural network^{［11，12］}，spatial attraction model^［13-16］，indicator cokriging^{［17，18］}，Hopfield neural network^{［19，20］}.

In addition，the interpolation-based subpixel mapping（ISPM）has been an important method of soft then hard type due to its simple physical meaning. The existing ISPM method basically contains two processing steps^［21］：1）interpolation and 2）class allocation. Firstly，the coarse fraction images are utilized to produce the fine fraction images with the predicted values（between 0 and 1）of all land cover classes for all subpixels by interpolation method. Then class allocation method is used to assign class labels to subpixel，producing the final mapping result. In particular，when selecting the bilinear interpolation（BI）or bicubic interpolation（BIC）as interpolation method，the mapping result can be very quickly achieved^{［21］-［28］}. However，due to the coarse resolution of the original image，the fraction images cannot pick up the full spatial-spectral information of the original image，the mapping result will be affected.

In this paper，using pansharpening technique improves interpolation-based subpixel mapping（PAN-ISPM）is proposed. The original coarse remote sensing image is first fused with the high resolution panchromatic image from the same area by pansharpening technique^［29］ in a novel processing path，and the improved image is unmixed to obtain the novel fine fraction images. The finer fraction images with more spatial-spectral information are then derived by integrating the two kinds of the fine fraction images from the novel path and the existing path. Finally，the predicted values from the finer fraction images are utilized to allocate hard class labels to all subpixels to achieve mapping result. The proposed method realizes that the more spatial-spectral information of the original image is supplied. Experimental results show that the proposed PAN-ISPM can obtain the better mapping result than the existing ISPM methods.

1 Method

1.1　ISPM model

Suppose $S$ is the zoom factor，the spectral unmixing results of the original coarse remote sensing image are $K$ （ $K$ is the number of land cover classes）coarse fraction images $L_{k}$ （ $k$ =1，2，…， $K$ ），and each mixed pixel is divided into $S \times S$ subpixels. Suppose $L_{k} (P_{J})$ is the fraction of the kth class for pixel $P_{J}$ （ $J$ =1，2，…， $M$ ， $M$ is the number of pixels）and $H_{k}^{1} (p_{j})$ is the predicted value for the $k$ th class at subpixel $p_{j}$ （ $j$ =1，2，…， $M S^{2}$ ， $M S^{2}$ is the number of subpixels）.

As shown in Fig. 1，taking the fraction images $L_{k}$ as inputs，the interpolation method is utilized to produce the fine fraction images，each of which is composed of MS² predicted values $H_{k}^{1} (p_{j})$ . Constraints from class fractions should meet the formula as follow：

N_{k} (P_{J}) = R o u n d (L_{k} (P_{J}) \times S^{2})

，（1）

where $N_{k} (P_{J})$ is the number of subpixels for the $k$ th class， $R o u n d (•)$ is a function that takes the integer nearest to $L_{k} (P_{J}) \times S^{2}$ .

Figure 1.The flowchart of ISPM

Finally，class allocation method is utilized to allocate the class labels to all subpixels according to the predicted values.

1.2　PAN-ISPM model

As shown in Fig. 1，we can find that the existing ISPM method is implemented in the coarse fraction images derived from the spectral unmixing. However，due to the coarse resolution of the original image，the spatial-spectral information of the original image is not fully utilized. To solve this problem，the PAN-ISPM is proposed. The flowchart of PAN-ISPM is shown in Fig. 2.

Figure 2.The flowchart of proposed PAN-ISPM

Firstly，the resolution of the original image is improved by pansharpening technique in a novel processing path. The main purpose of this paper is to improve the existing ISPM model by the new processing path. Pansharpening technique is just a tool to get new processing path. Therefore，we only consider the role of the new processing path. Due to effectively rendering spatial details and fast implementation，principal component analysis（PCA）is selected as the pansharpening method here. Other more effective pansharpening methods can also be used in the new path，but it is beyond the scope of this article. The novel fine fraction images with predicted values $H_{k}^{2} (p_{j})$ are derived by unmixing the improved image.

Secondly，the finer fraction images with the predicted values $F_{k}^{} (p_{j})$ are obtained by integrating the novel fine fraction images from the novel processing path and the existing fine fraction images from the existing processing path by the appropriate parameter $θ$ . Due to its simple physical meaning，bilinear interpolation ^［23］ is selected as the interpolation method in the existing processing path.

The formula of integrating is given as：

F_{k}^{} (p_{j}) = (1 - θ) H_{k}^{1} (p_{j}) + θ H_{k}^{2} (p_{j})

，（2）

Finally，class allocation method is utilized to obtain the mapping result according to the predicted values $F_{k}^{} (p_{j})$ from the finer fraction images. Linear optimization^［30］ is employed as class allocation method here.

Since the resolution of the original coarse image is improved by pansharpening technique，the more spatial-spectral information is supplied to improve the final mapping result.

2 Experiment

Five ISPM methods are tested and compared：bilinear interpolation（BI）^［22］，bicubic interpolation（BIC）^［23］，spatial-spectral bilinear interpolation（SS-BI）^［21］，hybrid interpolation based on parallel paths（HIPP）^［27］，and the proposed PAN-ISPM. The accuracy of mapping result is evaluated quantitatively by the percentage of correctly classified pixels（PCC）and Kappa coefficient（Kappa）^［5］. All experiments are tested on a Pentium（R）Dual-core Processor（2.20 GHz）with MATLAB R2018 version.

2.1　Simulated Data

The original fine remote sensing image is downsampled by $S \times S$ low pass filter to produce the simulated coarse image for quantitative assessment. Since the land cover classes at the subpixel level are known in the downsampled case，we can facilitate direct evaluation of the impact of image registration error on the technique. The original fine hyperspectral image performed on an urban site of the airborne HYDICE is from the mall in Washington DC. As shown in Fig. 3（a），the tested region is with 240×240 pixels and 191 spectral bands. As shown in Fig. 3（b），the coarse image is generated by degrading the fine data by $S = 2$ .

To avoid the effect of errors caused by the acquisition of the panchromatic image，only considering the effect of pansharpening technique，the spectral response of the IKONOS satellite is utilized in the original remote sensing image to create appropriate synthetic panchromatic image ^［31］. The panchromatic image is shown in Fig. 3（c）. As shown in Fig. 3（d），the pansharpening result is close to the original fine image. The appropriate parameter $θ$ is selected as 0.5.

Figure 3.(a) False color image of Washington DC (bands 65, 52, and 36 for red, green, and blue, respectively). (b) Coarse image ( $S = 2$ ). (c) Panchromatic image. (d) Pansharpening result.

As shown in Fig. 4（a），the reference image is obtained from the Fig. 3（a）by classification method，including shadow，water，road，tree，grass，roof and trail. The mapping results of the five methods are shown in Figs. 4（b）-（f）. A visual comparison of the results suggests that the proposed PAN-ISPM model is closer to the reference map due to utilizing more spatial-spectral information.

Figure 4.(a) Reference image, (b) BI, (c) BIC, (d) SS-BI, (e) HIPP, (f) PAN-ISPM

Five ISPM methods are quantitatively evaluated by the classification accuracy of each class，PCC and Kappa. Checking the Table 1，the accuracy of PAN-ISPM is superior to the existing ISPM methods. With respect to the overall accuracy，PAN-ISPM increases the PCC by around 5.6% when compared with HIPP. PAN-ISPM obtains the highest Kappa of 0.842 6.

Class	BI	BIC	SS-BI	HIPP	PAN-ISPM
Shadow	73.44%	75.03%	81.62%	82.83%	86.28%
Water	85.56%	88.97%	94.45%	94.73%	95.03%
Road	70.55%	72.74%	76.29%	79.10%	81.16%
Tree	72.45%	75.45%	76.61%	78.64%	79.14%
Grass	74.70%	78.60%	82.74%	83.93%	86.60%
Roof	70.67%	72.98%	77.18%	78.56%	80.02%
Trail	73.88%	75.58%	79.08%	82.53%	84.01%
PCC	76.82%	77.47%	80.72%	81.54%	87.18%
Kappa	0.7356	0.7429	0.7772	0.8055	0.8426

Table 1. Accuracy evaluation of the five methods.

View all Tables

To evaluate the effect of the zoom factor $S$ on the performance of the results，the five methods are tested for the two other zoom factors of 4 and 6. The PCC and Kappa of the five methods for all three zoom factors are shown in Figs. 5（a）-（b）. We can note that as $S$ increases，the PCC and the Kappa of the five methods decrease. But consistent with the result presented in Table 1，the proposed PAN-ISPM produces the higher PCC and Kappa than the other four methods.

Figure 5.(a) PCC (%) of the five methods in relation to zoom factor $S$ , (b) Kappa of the five methods in relation to zoom factor $S$ .

2.2　Real data

To better demonstrate the effectiveness of the proposed PAN-ISPM，a real data set is used in experiment 2. A 30-m hyperspectral image is captured by the Hyperion satellite over Rome，Italy. As shown in Fig. 6（a），the tested region is with 300×160 pixels and 198 bands. As shown in Fig. 6（b），the 15-m panchromatic image is obtained from the Landsat 8 panchromatic band over the same area. This corresponds to the scale $S = 2$ between the original coarse remote sensing and the panchromatic image. The pansharpening result is shown in Fig. 6（c）. The appropriate parameter $θ$ is selected as 0.6.

Figure 6.(a) False color image of Rome (bands 150, 10, and 24 for red, green, and blue, respectively), (b) Panchromatic image, (c) Pansharpening result.

As shown in Fig. 7（a），the reference image includes vegetation，soil，built-up，and water. Figs. 7（b）-（f）show the ISPM results for the five methods. In Figs. 7（b）-（e），there are many speckle artifacts and disconnected holes in the land cover classes. The proposed PAN-ISPM produces more continuous and smoother land cover classes in Fig. 7（f）. Because more spatial-spectral information is utilized，the mapping result of PAN-ISPM is closer to the reference image. Table 2 lists the accuracy of each class（%），PCC（%），and Kappa of the five methods. Similar to the previous experimental results in experiment 1，the PAN-ISPM outperforms the other four ISPM methods.

Class	BI	BIC	SS-BI	HIPP	PAN-ISPM
Vegetation	60.08%	61.96%	66.56%	74.50%	75.93%
Soil	60.22%	61.43%	64.93%	65.80%	71.78%
Built-up	81.32%	82.42%	83.97%	84.99%	87.09%
Water	37.18%	44.10%	49.49%	54.36%	61.03%
PCC	70.62%	72.06%	74.89%	77.55%	80.03%
Kappa	0.587 7	0.598 5	0.616 4	0.639 9	0.673 6

Table 2. Accuracy evaluation of the five methods.

View all Tables

Figure 7.(a) Reference image, (b) BI, (c) BIC, (d) SS-BI, (e) HIPP, (f) PAN-ISPM

2.3　Discussion

First，the weight parameter $θ$ is introduced to balance the influence of $H_{k}^{1} (p_{j})$ and $H_{k}^{2} (p_{j})$ on the PAN-ISPM. Here we choose the appropriate parameter value through multiple tests. Experiment 1 and 2 are repeated to evaluate the PCC（%）for ten combinations of θ in the range of［0，0.9］at an interval of 0.1 in order to determine the most suitable value of $θ$ . As shown in Fig. 8，it can be seen that the most appropriate $θ$ value of Experiments 1 and 2 are 0.5 and 0.6，respectively，the most appropriate value is the one at which the PCC（%）is the highest.

Figure 8.PCC (%) of the two experiments in relation to weight parameter $θ$ .

Second，the computing time is an important index to estimate the performance of ISPM methods. The computing time of five ISPM methods in experiment 1 and 2 is shown in Fig. 9. Because the PAN-ISPM includes processing step of pansharpening，the proposed method is complex and time consuming compared to the existing ISPM methods. This is the cost of obtaining higher mapping precision.

Figure 9.Computing time of the five ISPM methods in the two experiments

Finally，the performance of PAN-ISPM depends on pansharpening technique. Therefore，it is necessary to test the effects of different pansharpening methods on the performance of the proposed method. The band-dependent spatial detail（BDSD）^［31］ is selected as another pansharpening method to compare the previous PCA in the experiment 1 and 2. Fig. 10 show the PCC（%）of PAN-ISPM result in relation to two pansharpening methods. As shown in Fig. 10，since BDSD is more effective than PCA，the PCC（%）in BDSD-based PAN-ISPM is higher than that in PCA-based PAN-ISPM. Hence，the more effective pansharpening method can obtain the better mapping result.

Figure 10.PCC (%) of PAN-ISPM result in relation to BDSD and PCA in the two experiments

3 Conclusion

In this paper，the PAN-ISPM is proposed to improve the mapping result. First of all，the original coarse hyperspectral image is utilized to obtain the improved image by pansharpening in the novel processing path，and the improved image is unmixed to produce the novel fine fraction images. The finer fraction images with more spatial-spectral information e then obtained by integrating the novel fine fraction images from the novel path and the existing fine fraction images from the existing path. Finally，the final mapping result is derived by class allocation method according to the predicted values from the finer fraction images. Because the coarse resolution of the original image is improved by pansharpening in the novel processing path，the more spatial-spectral information of the original image could be fully supplied to ISPM，and the final mapping result is improved. The visual and quantitative comparison with the existing ISPM methods shows the result of the PAN-ISPM is better.

The appropriate parameter $θ$ is selected by multiple tests in this paper. Therefore，an adaptive method for selecting $θ$ is worth studying in future work. In addition，the PAN-ISPM includes more processing steps than the other four ISPM methods. Therefore，it is necessary to optimize the structure of the proposed method and speed up its operation in the future.

References

[1] Feng LING, Shen-Jun WU, Fei XIAO et al. Sub-pixel mapping of remotely sensed imagery: a review. Journal of Image and Graphics, 16, 1335-1345(2011).

[2] F Ling, G M Foody. Super-resolution land cover mapping by deep learning. Remote Sensing Letters, 10, 598-606(2019).

[3] Q Wang, C Zhang, P M Atkinson. Sub-pixel mapping with point constraints. Remote Sensing of Environment, 224, 111817(2020).

[4] Q M Wang, W Shi, L Wang. Allocating classes for soft-then-hard sub-pixel mapping algorithms in units of class. IEEE Transactions on Geoscience and Remote Sensing, 52, 2940-2959(2014).

[5] P M Atkinson. Sub-pixel target mapping from soft-classified remotely sensed imagery. Photogrammetric Engineering and Remote Sensing, 71, 839-846(2005).

[6] A Villa, J Chanussot, J A Benediktsson et al. Spectral unmixing for the classification of hyperspectral images at a finer spatial resolution. IEEE Journal of Selected Topics in Signal Processing, 5, 521-533(2011).

[7] F Ling, W Li, Y Du et al. Land cover change mapping at the subpixel scale with different spatial-resolution remotely sensed imagery. IEEE Geoscience and Remote Sensing Letters, 8, 182-186(2011).

[8] Y Makido, A Shortridge, J P Messina. Assessing alternatives for modeling the spatial distribution of multiple land-cover classes at subpixel scales. Photogrammetric Engineering and Remote Sensing, 73, 935-943(2007).

[9] D He, Y Zhong, R Feng et al. Spatial-temporal sub-pixel mapping based on swarm intelligence theory. Remote Sensing, 894(2016).

[10] X Tong, X Xu, A Plaza. A new genetic method for subpixel mapping using hyperspectral images. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 9, 4480-4491(2016).

[11] D Nigussie, R Zurita-Milla, J G P W Clevers. Possibilities and limitations of artificial neural networks for subpixel mapping of land cover. International Journal of Remote Sensing, 32, 7203-7226(2011).

[12] Xiong XU, Yan-Fei ZHONG, Liang-Pei ZHANG et al. A sub-pixel mapping algorithm based on BP neural network with spatial autocorrelation function for remote sensing imagery. Acta Geodaetica et Cartographica Sinica, 40, 307-311(2011).

[13] F Ling, X Li, Y Du et al. Sub-pixel mapping of remotely sensed imagery with hybrid intra- and inter-pixel dependence. International Journal of Remote Sensing, 34, 341-357(2013).

[14] P Wang, L Wang. Soft-then-hard super-resolution mapping based on a spatial attraction model with multiscale sub-pixel shifted images. International Journal of Remote Sensing, 38, 4303-4326(2017).

[15] L Lu, Y Hang, L Di. A new spatial attraction model for improving subpixel land cover classiﬁcation. Remote Sensing, 360(2017).

[16] P Wang, L Wang, H Leung et al. Super-resolution mapping based on spatial-spectral correlation for spectral imagery. IEEE Transactions on Geoscience and Remote Sensing(2020).

[17] H Jin, G Mountrakis, P Li. A super-resolution mapping method using local indicator variograms. International Journal of Remote Sensing, 33, 7747-7773(2012).

[18] Q Wang, P M Atkinson, W Shi. Indicator cokriging-based subpixel mapping without prior spatial structure information. IEEE Transactions on Geoscience and Remote Sensing, 53, 309-323(2015).

[19] X Li, Y Du, F Ling et al. Superresolution mapping of remotely sensed image based on hopfield neural network with anisotropic spatial dependence model. IEEE Geoscience and Remote Sensing Letters, 11, 1265-1269(2014).

[20] P Wang, L Wang, H Leung et al. Subpixel mapping based on hopfield neural network with more prior information. IEEE Geoscience and Remote Sensing Letters, 16, 1284-1288(2019).

[21] P Wang, L Wang, J Chanussot. Soft-then-hard subpixel land cover mapping based on spatial-spectral interpolation. IEEE Geoscience and Remote Sensing Letters, 13, 1851-1854(2016).

[22] L Wang, Z Wang, Z Dou et al. Edge-directed interpolation-based sub-pixel mapping. Remote Sensing Letters, 12, 1195-1203(2013).

[23] W Wang Q and Shi. Utilizing multiple subpixel shifted images in subpixel mapping with image interpolation. IEEE Geoscience and Remote Sensing Letters, 11, 798-802(2014).

[24] P Wang, L Wang, M D Mura et al. Using multiple subpixel shifted images with spatial-spectral information in soft-then-hard subpixel mapping. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 10, 2950-2959(2017).

[25] Q Wang, W Shi, P M Atkinson. Sub-pixel mapping of remote sensing images based on radial basis function interpolation. ISPRS Journal of Photogrammetry and Remote Sensing, 92, 1-15(2014).

[26] Y. Chen, Y. Ge, D. Song. Superresolution land-cover mapping based on high-accuracy surface modeling. IEEE Transactions on Geoscience and Remote Sensing, 12, 2516-2520(2015).

[27] P Wang, G Zhang, Y Kong et al. Superresolution mapping based on hybrid interpolation by parallel paths. Remote Sensing Letters, 10, 149-157.

[28] G Vivone, L Alparone, J Chanussot et al. A critical comparison among pansharpening algorithms. IEEE Transactions on Geoscience and Remote Sensing, 53, 2565-2585(2015).

[29] J. Verhoeye, R. De Wulf. Land-cover mapping at sub-pixel scales using linear optimization techniques. Remote Sensing of Environment, 79, 96-104(2002).

[30] T. M. Tu, P. S. Huang, C. L. Hung, C. P. Chang. A fast intensity-hue-saturation fusion technique with spectral adjustment for IKONOS imagery. IEEE Geoscience and Remote Sensing Letters, 1, 309-312(2004).

[31] A Garzelli, F Nencini, L Capobianco. Optimal MMSE pan sharpening of very high resolution multispectral images. IEEE Transactions on Geoscience and Remote Sensing, 46, 228-236(2008).