Image Semantic Segmentation Based on Multi-Scale Feature Extraction and Fully Connected Conditional Random Fields

Yongfeng Dong; Yuxin Yang; Liqin Wang

doi:10.3788/LOP56.131007

Journals >Laser & Optoelectronics Progress >Volume 56 >Issue 13 >Page 131007 > Article

Laser & Optoelectronics Progress
Vol. 56, Issue 13, 131007 (2019)

Image Semantic Segmentation Based on Multi-Scale Feature Extraction and Fully Connected Conditional Random Fields

Yongfeng Dong^1、2, Yuxin Yang¹, and Liqin Wang^1、2、*

Author Affiliations

¹ School of Artificial Intelligence, Hebei University of Technology, Tianjin 300401, China

² Hebei Provincial Key Laboratory of Big Data Computing, Tianjin 300401, China

show less

DOI: 10.3788/LOP56.131007 Cite this Article Set citation alerts

Yongfeng Dong, Yuxin Yang, Liqin Wang. Image Semantic Segmentation Based on Multi-Scale Feature Extraction and Fully Connected Conditional Random Fields[J]. Laser & Optoelectronics Progress, 2019, 56(13): 131007 Copy Citation Text

show less

Fig. 1. Single-branch network structure before feature fusion

Download full size

Fig. 2. Multi-scale feature fusion

Download full size

Fig. 3. FullCRF optimization semantic rough segmentation result

Download full size

Fig. 4. Comparison of segmentation results

Download full size

RGB encoder			Depth encoder
Conv block1:3×3 Conv 643×3 Conv 642×2 maxpooling	Conv block2:3×3 Conv 1283×3 Conv 1282×2 maxpooling	Conv block3:3×3 Conv 2563×3 Conv 2562×2 maxpooling	Conv block1:3×3 Conv 643×3 Conv 642×2 maxpooling	Conv block2:3×3 Conv 1283×3 Conv 1282×2 maxpooling	Conv block3:3×3 Conv 2563×3 Conv 2562×2 maxpooling
Conv block4:3×3 Conv 5123×3 Conv 5123×3 Conv 5122×2 maxpooling	Conv block5:3×3 Conv 5123×3 Conv 5123×3 Conv 5122×2 maxpooling		Conv block4:3×3 Conv 5123×3 Conv 5123×3 Conv 5122×2 maxpooling	Conv block5:3×3 Conv 5123×3 Conv 5123×3 Conv 512

Table 1. Parameter setting table of single branch encoder before feature fusion

Method	Inputdata type	PA /%	MA /%	MIoU /%
Method in Ref. [6]	RGB	60.0	42.2	29.2
Method in Ref. [6]	Depth	57.1	35.2	24.2
Method in Ref. [25]	RGB-depth	60.3	-	28.6
Method in Ref. [26]	RGB-depth	63.8	31.5	-
Method in Ref. [6]	RGB-depth	61.5	42.4	30.5
Method in Ref. [22]	RGB-depth	65.6	42.2	27.8
MSF-CRF	RGB-depth	66.9	44.2	30.2

Table 2. Results of different networks on NYUv2 dataset

Dataset	Wall	Floor	Cabinet	Bed	Chair	Sofa	Table	Door
FuseNet	89.2	95.7	67.9	75.7	74.6	71.0	49.3	34.8
MSF-CRF	91.8	96.5	71.0	73.7	73.5	83.1	49.5	27.1
Dataset	Window	Bookshelf	Picture	Counter	Blinds	Desk	Shelf	Curtain
FuseNet	52.9	48.0	68.1	56.4	67.2	15.1	12.6	56.5
MSF-CRF	53.8	60.7	66.6	63.5	45.6	26.0	17.3	58.5
Dataset	Dresser	Pillow	Mirror	Floormat	Clothes	Ceiling	Books	Fridge
FuseNet	28.4	44.3	30.7	38.8	22.9	75.5	21.2	11.9
MSF-CRF	45.3	49.3	54.9	19.0	15.9	69.2	10.7	21.0
Dataset	TV	Paper	Towel	Shower	Box	White board	Person	Nightstand
FuseNet	39.1	5.7	23.0	34.9	7	32.5	23.2	35.1
MSF-CRF	50.8	4.3	29.6	30.6	3.3	24.3	49.4	54.0
Dataset	Toilet	Sink	Lamp	Bathtub	Bag	Other struct	Other furniture	Other prop
FuseNet	75.0	32.4	40.1	51.9	1.6	19.8	10.8	45.7
MSF-CRF	78.7	32.9	40.2	50.1	1.0	9.3	18.7	46.8

Table 3. Comparison of classification accuracy of 40 categories

Dataset	Wall	Floor	Cabinet	Bed	Chair	Sofa	Table	Door
FuseNet	59.5	70.8	44.7	59.3	41.2	47.5	31.8	19.6
MSF-CRF	57.2	70.4	45.0	63.7	43.8	50.2	35.4	15.4
Dataset	Window	Bookshelf	Picture	Counter	Blinds	Desk	Shelf	Curtain
FuseNet	27.5	30.0	44.1	34.4	42.5	11.3	5.8	34.8
MSF-CRF	32.7	30.8	48.0	38.5	36.3	17.0	6.1	43.1
Dataset	Dresser	Pillow	Mirror	Floormat	Clothes	Ceiling	Books	Fridge
FuseNet	23.7	29.6	24.3	29.5	8.5	42.3	14.8	8.9
MSF-CRF	32.1	34.3	42.5	17.0	9.4	39.8	9.5	14.0
Dataset	TV	Paper	Towel	Shower	Box	White board	Person	Nightstand
FuseNet	31.5	3.8	18.5	20.3	4	22.4	14.8	26.6
MSF-CRF	39.1	3.7	21.8	26.1	2.4	20.7	32.9	40.1
Dataset	Toilet	Sink	Lamp	Bathtub	Bag	Other struct	Other furniture	Other prop
FuseNet	49.1	24.3	28.8	41.1	1.1	11.1	7.9	21.9
MSF-CRF	50.1	21.2	31.2	39.8	0.9	7.3	13.4	25.0

Table 4. Comparison of IoU of 40 categories

Download Citation

Set citation alerts for the article

Tools

Set citation alerts for the article

Save the article for my favorites

Paper Information