Color night vision technology is able to fuse the dual-spectral image of low-light-level and infrared image into a color one suited to human observation. Furthermore, a appropriate scene parsing method on the color night vision image could additionally facilitate human observation by providing automatic content analysis. An online scalable scene parsing method was proposed aiming at the rich and changeable color night vision in practice which required algorithms with high flexibility. The proposed method was based on a non-parametric model that needed no training process when predicting scene categories. It matched the query image and the sample images in database using both global and local features, and then transfered semantic labels of the best-match samples to the query image. Moreover, the database can be dynamically expanded according to different usage scenarios. The experimental results show that proposed method achieves satisfactory accuracies on color night vision images that obtained by a variety of color night vision methods, including the statistical color mapping, TNO, and NRL, throughout diverse scenes, including cities, countryside and others.Markov random field