The selection of suitable hyperparameters in wavefront sensorless adaptive optics systems is the key to achieve the best performance of iterative control algorithms. Existing iterative control algorithms for hyperparameter setting generally use the traversal method, which is easy to understand and implement, but is computationally intensive and time-consuming, and may also miss the global optimum because of finding a local optimum. A Bayesian optimization method was adopted for selecting hyperparameters suitable for iterative control algorithms of adaptive optics systems. The commonly-used stochastic parallel gradient descent algorithm (SPGD), Momentum-SPGD and CoolMomentum-SPGD control algorithms were used as examples to compare and analyze the calibration effects of control algorithms using the traversal method and Bayesian optimization method to select hyperparameters, respectively. The results show that the advantages of using Bayesian optimization method for hyperparameter selection were obvious. For the SPGD control algorithm, the number of sample instances required for the Bayesian optimization method is 10% of that for the traversal method when the same convergence effect is achieved, and for the Momentum-SPGD and CoolMomentum-SPGD control algorithms, the number of sample instances required for the Bayesian optimization method is 7% and 9% of that for the traversal method, respectively. The above findings can provide a theoretical basis for hyperparameter setting in the practical application of iterative control algorithms for adaptive optical systems.