Bayesian optimization has risen over the last few years as a very attractive approach to find the optimum of noisy, expensive to evaluate, and possibly black-box functions. One of the fields where these functions are common is in machine-learning, where one typically has to fit a particular model by minimizing a specified form of loss. In this Master s thesis we first focus on reviewing the most recent literature on Gaussian Processes as well as Bayesian optimiza- tion methods, then we benchmark said methods against several real case machine-learning scenarios and lastly we provide open source software that will allow researchers to apply these strategies in other problems.