# Hyperparameter Optimization With Hyperopt — Intro & Implementation | by Farzad Mahmoodinobar | Jun, 2023

## 2.1. Support Vector Machines and Iris Data Set

In a previous post I used Grid Search, Random Search and Bayesian Optimization for hyperparameter optimization using the Iris data set provided by scikit-learn. Iris data set includes 3 different irises petal and sepal lengths and is a commonly-used data set for classification exercises. In this post, we will use the same data set but we will use a Support Vector Machine (SVM) as a model with two parameters that we can optimize as follows:

• `C`: Regularization parameter, which trades off misclassification of training examples against simplicity of the decision surface.
• `gamma`: Kernel coefficient, which defines how much influence a single training example has. The larger gamma is, the closer other examples must be to be affected.

Since the goal of this exercise is to go through the hyperparameter optimization, I will not go deeper into what SVMs do but if you are interested, I find this scikit-learn post helpful.

We will generally follow the same steps that we used in the simple example earlier but will also visualize the process at the end:

1. Import necessary libraries and packages
2. Define the objective function and the search space
3. Run the optimization process
4. Visualize the optimization

## 2.1.1. Step 1 — Import Libraries and Packages

Let’s import the libraries and packages and then load the data set.

`# Import libraries and packagesfrom sklearn import datasetsfrom sklearn.svm import SVCfrom sklearn.model_selection import cross_val_score# Load Iris datasetiris = datasets.load_iris()X = iris.datay = iris.target`

## 2.1.2. Step 2 — Define Objective Function and Search Space

Let’s first start with defining the objective function, which will train an SVM and returns the negative of the cross-validation score — that is what we want to minimize. Note that we are minimizing the negative of cross-validation score to be consistent with the general goal of “minimizing” the objective function (instead of “maximizing” the cross-validation score).

`def objective_function(parameters):clf = SVC(**parameters)score = cross_val_score(clf, X, y, cv=5).mean()return -score`

Next we will define the search space, which consists of the values that our parameters of `C` and `gamma` can take. Note that we will use Hyperopt’s `hp.uniform(label, low, high)`, which returns a value uniformly between “low” and “high” (source).

`# Search Spacesearch_space = {'C': hp.uniform('C', 0.1, 10),'gamma': hp.uniform('gamma', 0.01, 1)}`

## 2.1.3. Run Optimization

Same as the simple example earlier, we will use a TPE algorithm and store the results in a `Trials` object.

`# Trials object to store the resultstrials = Trials()# Run optimizationbest = fmin(fn=objective_function, space=search_space, algo=tpe.suggest, trials=trials, max_evals=100)`

Results: