Based on the full convolutional network, an image segmentation model is proposed to obtain the target segmentation results. This model consists of two deep neural network branches with the same structures. As for each branch, a convolution-deconvolution structure is adopted to implement the feature extraction and to recover the target area from the features. These two branches receive different image inputs, and then the final segmentation results are obtained via the weighted fusion of the results from these two branches. This model combines the multi-level scale features of different image sources, and the training model is more robust through data enhancement when the number of training samples is limited. The experiments are carried out on the optical image dataset of Weizmann horse and the remote sensing image dataset of Vaihigen. The comparison with the related literatures is also made. The results show that the proposed model has a higher target segmentation integrity and an optimal segmentation performance. The ideal extraction results of remote sensing image buildings under the conditions of limited training data, various shapes, large scale changes and so on, indicate that the proposed model can be applied to the complex remote sensing image object segmentation.