(Y0.96Er0.02Yb0.02)O3 nanocrystals of 10 and 40 nm average particle size were prepared by combustion method. And bulk materials of the same components were obtained by annealing at 1 200 ℃. X-ray diffraction (XRD), Fourier transform infrared (FTIR) spectra, transmission electron microscope (TEM), and scanning electron microscopy (SEM) were used to characterize the crystal structure and morphology of the samples. The upconversion emission spectra and NIR (near-infrared) emission spectra were measured, under 980 nm excitation. The research result indicates that as the particle size decreases, the upconversion red emission and NIR emission components increase in the emission spectra. This phenomenon is attributed to the large ratio of surface area to volume in nanocrystals. This characteristic makes the nanocrystals absorb more OH-, whose vibrational energy is 3 200-3 800 cm-1. The increase in the OH－ number enhances the rate of nonradiative relaxation from Er3+ 4I11/2 to 4I13/2 energy level (energy gap is 3 600 cm-1). This nonradiative relaxation process depopulates the 4I11/2 level and makes the green emission weaker. Meanwhile, this process populates the 4I13/2 level and makes the red and NIR emissions stronger. The intensity of 1.5 μm main peak is 1.6 times that of bulk materials. This result has great significance in actual applications of nanophosphors.