Near-infrared (NIR) spectroscopy rapid detection technology was used to determine protein content in instant coffee. Support vector machine (SVM) and extreme learning machine (ELM) was applied for validating their practicality in modeling analysis. We proposed the latent variable SVM (LV-SVM) and latent variable ELM (LV-ELM) methods combined with latent variable analysis technique. Thetuning of latent variables and the optimization of the key parameters in machines were joint in-one so that the data dimension reduction and the selection of machine parameters can be both accomplished in one single modeling process. The calibrating-validating-testing mechanism was used for sample division. The NIR analytical models were trained based on the calibrating sample set. The model prediction results were generated and saved as a 3D box as they were determined by the simultaneous tuning of the latent variable and the machine parameter. Then the joint optimization of model parameters was selected in the way of predicting the validating samples. Further, the optimal model was evaluated by the testing samples. The optimal LV-SVM model gave the validating root mean square error as 6.797; the corresponding testing root mean square error as 8.384. The optimal LV-ELM model obtained the validating root mean square error as 6.118. The corresponding testing root means square error as 7.837. Compared with the common partial least square method, the LV-SVM and LV-ELM methods have better prediction results, which verified the application advantages of the latent variable machine learning method in near-infrared quantitative analysis. This proposed method is expected for further application in content detection of other kinds of coffee.