Image fusion at pixel level without precise registration always causes wrong colors and other problem. Classification-based fusion scheme can effectively eliminate the incorrect color at the edge of the objective. But the traditional per-pixel classification results in the well-known salt and pepper effect. The only way to smooth the image is to use filters, and it would impact on the result of fusion. A method consisted of a sequential application of segmentation, classification and fusion techniques is proposed. Firstly, the object-oriented classification was used. And it can eliminate the influence of salt and pepper effect. Subsequently, according to the restriction of the precise classification result, adjusting the multi-spectral image, then the fusion by using Hue-Saturation-Value (HSV) color transformation. Finally, this merging technique was compared to other merging methods. The quantitative comparison result showed that it was not only provided a better visual representation but also had higher value in the variance and entropy.