The chlorophyll and carotenoid content is an important indicator for evaluating the health status of plants. The PROSPECT model, coupled with machine learning, has been widely used to retrieve the biochemical properties of vegetation. However, the application of the coupled model is limited due to the differences between the leaf-directional hemispherical reflectance factor (DHRF) spectra and the bidirectional reflectance factor (BRF) spectra. This paper utilizes the leaf spectral database of North American plant (EcoSIS) as the experimental dataset and introduces the PROSPECT model as an additional constraint for machine learning. This approach creates a hybrid dataset by employing wavelet continuous wavelet transform (CWT) to generate the wavelet coefficient spectrum and the derivative spectrum generated by the first-order derivative (FD). Three kinds of feature extraction algorithms, namely competitive adaptive reweighting algorithm (CARS), successive projection algorithm (SPA), and principal component analysis (PCA) were applied to extract spectral features for chlorophylls and carotenoids in the full-spectral domains and the subdomain of VNIR spectroscopy. Based on the above 12 combinations of different methods, artificial neural network (ANN) prediction models for chlorophyll and carotenoids were separately established. The results show that the simulated data under the constraint of the PROSPECT model enhanced the quality of the training set for machine learning to a certain extent. Additionally, the spectra processed by the first-order derivatives and wavelet transforms were able to reduce better the bias between the simulated spectra of the DHRF and the measured spectra of the BRF. The best inversion of leaf chlorophyll is achieved with the FD+CARS combination in the whole spectral domain, yielding a test set R2 of 0.806 4 and RMSE of 2.911 4. Meanwhile, the CWT+CARS combination in the VNIR spectral sub-domain offers the best results for leaf carotenoids, with a test set R2 of 0.797 2 and RMSE of 0.414 1. The proposed method can provide researchers with a reference to extract biochemical characteristics of plant leaves more accurately and efficiently from BRF spectra and other near-end reflectance images.