Wood density is an important physical property of wood which can reflect a variety of physical properties such as wood shrinkage, compressive and tensile strength. Using near-infrared spectroscopy technology can rapidly predict wood density, which can overcome the disadvantages of traditional detection methods that consume workforce, material resources and time. However, the modeling results are often affected by abnormal samples. In order to accurately identify and eliminate abnormal samples in the sample set, an isolation forest combined with the studentized residual method (IFSR) was proposed. Based on the advantages of integrated features of isolated forests, the influence of samples on the model is considered, and abnormal samples and strong influence samples can be detected simultaneously. This study measured the near-infrared spectra of 181 larch wood samples and their air-dry density at room temperature. By comparing a variety of preprocessing and feature selection methods, the preprocessing method was determined to adopt the standard normal variable change (SNV) + detrending processing (DT) + mean centralization (MC) + standardization (Auto) method and the feature wavelength selection was determined to adopt competitive adaptive reweighted sampling (CARS) method. Eliminated the influence of noise and irrelevant information on the algorithm, simplified the dataset, and improved the algorithm's accuracy in removing abnormal samples. In order to verify the ability of the IFSR method to eliminate abnormal samples, it was compared with the other six anomaly detection methods such as Monte Carlo Interactive Verification (MCCV), Mahalanobis Distance (MD), etc. The partial least squares (PLS) model was established to evaluate its anomaly detection performance. At the same time, the particle swarm optimization-support vector machine regression (PSO-SVR), BP neural network (BPNN) and PLS were used to establish the near-infrared prediction model of larch wood density. The results show that the optimized model obtained by IFSR combined with PSO-SVR has the strongest predictive ability, and IFSR can effectively eliminate singular samples and improve the model's accuracy.