Aiming to solve problems of the weak ability on adaptation and noise resistance in object segmentation, a novel PCNN based moving object segmentation method is presented in H.264/AVC compressed domain. First, a spatial-temporal vector filtering is used as the preprocessor to reduce the target loss rate. Then, a forward-backward vector cumulative method is proposed to enhance the reliability of motion vectors. Finally, a Fusion-PCNN model is designed to fuse the cumulative motion field and the macro-block coding mode, which enhances the ability of noise resistance in object segmentation and limits the complexity. Moreover, the minimum cross-entropy is used to determine the firing conditions for an optimal self-adaptive segmentation template. Experimental results show that the proposed algorithm is outperformance and has the ability of self-adaptation and noise resistance in object segmentation. More accurate results are presented by the surveillance video.