There are many valuable rare and unusual objects in spectra dataset of Sloan Digital Sky Survey (SDSS) Data Release eight (DR8), such as special white dwarfs (DZ, DQ, DC), carbon stars, white dwarf main-sequence binaries (WDMS), cataclysmic variable (CV) stars and so on, so it is extremely significant to search for rare and unusual celestial objects from massive spectra dataset. A novel algorithm based on Kernel dense estimation and K-nearest neighborhoods (KNN) has been presented, and applied to search for rare and unusual celestial objects from 546 383 stellar spectra of SDSS DR8. Their densities are estimated using Gaussian kernel density estimation, the top 5 000 spectra in descend order by their densities are selected as rare objects, and the top 300 000 spectra in ascend order by their densities are selected as normal objects. Then, KNN were used to classify the rest objects, and simultaneously K nearest neighbors of the 5 000 rare spectra are also selected as rare objects. As a result, there are totally 21 193 spectra selected as initial rare spectra, which include error spectra caused by deletion, redden, bad calibration, spectra consisting of different physically irrelevant components, planetary nebulas, QSOs, special white dwarfs (DZ, DQ, DC), carbon stars, white dwarf main-sequence binaries (WDMS), cataclysmic variable (CV) stars and so on. By cross identification with SIMBAD, NED, ADS and major literature, it is found that three DZ white dwarfs, one WDMS, two CVs with company of G-type star, three CVs candidates, six DC white dwarfs, one DC white dwarf candidate and one BL Lacertae (BL lac) candidate are our new findings. We also have found one special DA white dwarf with emission lines of CaⅡ triple and MgⅠ, and one unknown object whose spectrum looks like a late M star with emission lines and its image looks like a galaxy or nebula.