Near-infrared diffuse reflectance spectroscopy has many advantages, such as being non-invasive, continuous, non-infectious, fast, in the non-invasive detection of body components. It has a great prospect in the application of blood glucose measurement in vivo. However, outliers often occur in the process of measurement due to the random noise, the change in interference components or the measurement conditions. Therefore, it is of great significance to eliminate the outliers in the near-infrared spectroscopy and thus improve the reliability of non-invasive blood components measurement. In this paper, the types of outliers that may occur in the blood glucose sensing by near-infrared diffuse reflectance were analyzed, and a three-dimensional coordinate outlier determination method based on the three-dimensional space constructed by the residual of chemical value, the Mahalanobis distance and the spectral residuals was proposed firstly. Then, it was used to discriminate the outliers in the simulated spectra of three-layer skin model by Monte Carlo program, where the abnormal data was obtained by adding the artificial errors, abnormal chemical values and abnormal temperature changes in the parameters setting in Monte Carlo simulation. All the outliers could be found successfully by the three-dimensional coordinate outlier determination method, and the root-mean-square error of cross-validation (RMSECV) of the Partial Least Square (PLS) model was reduced from 212 to 11 mmol·L-1 after the removal of outliers. Further, the oral glucose tolerance tests (OGTTs) of three volunteers were carried out, where three groups of experimental data were obtained by measuring the reference blood glucose concentrations and collecting the diffuse reflectance of finger synchronously, and Monte Carlo Cross-Validation outlier detection method and three-dimensional coordinate method were used to detect the outliers, respectively. Results showed that, after the removal of outlier by the three-dimensional coordinate method, the coefficient of determination of calibration model increased significantly, and the average RMSECV value of calibration model for three sets of samples was reduced from 21 to 08 mmol·L-1, which was better than that of MCCV method. All these results indicated that, three-dimensional coordinate method can effectively determine the outlier in the near-infrared diffuse reflectance and it’s more suitable for the non-invasive blood glucose measurement in vivo by near-infrared diffuse reflectance spectroscopy.