# Hyperparameter Tuning

Searching over settings that are not learned from data — learning rate, depth, regularisation strength — using grid, random or model-based search.

**Rests on:** Gradient Descent

**Leads to:** Feature Selection

*Mathematical Foundations · Optimization*


## Why learn this

You build systems that make predictions — loan approvals, image classifications, text summaries — and those systems only work if their internal settings are correct. Without adjusting these settings, a model might perform far below its potential or fail completely. Tuning is the practical step that turns a prototype into something reliable enough to put in front of users. This is why training runs cost so much: finding the right configuration takes time and compute. Understanding this process explains why a model that worked perfectly on a tutorial dataset fails on your own data, and how you can fix it by systematically exploring different configurations rather than guessing.


## The intuition

Imagine you are adjusting the controls on a complex machine to get the best output. These controls are not changed during operation; you set them once before starting. In machine learning, these controls are called hyperparameters. For example, the 'learning rate' determines how quickly the model updates its knowledge after each example. If you set it too high, the model overcorrects and never settles on a good answer. If you set it low, it learns so slowly that you might wait days for one small improvement. The 'depth' of the model is another control: too shallow, and it cannot understand complex patterns; too deep, and it memorizes every detail of the training data, including the noise, making it useless on new cases. Tuning is the systematic process of testing different combinations of these pre-set controls to find the specific mix that yields the highest accuracy on unseen data.


## An analogy

Think of hyperparameter tuning like finding the perfect radio station frequency. You have a dial (the hyperparameters) that you can turn to scan the spectrum. Each position produces a signal strength (the model's performance). You want the spot where the signal is clearest and loudest. Because you cannot see the whole spectrum at once, you might try turning the dial in broad steps first to find the general area with strong signals, then make finer adjustments within that area to pinpoint the exact best frequency. This avoids trying every single microscopic setting on the entire radio dial, which would take too long. Caveat: unlike a radio where the signal quality is instantly visible, evaluating a machine learning model takes significant time and effort for each 'turn' of the dial.


## Definition

Hyperparameter tuning is the process of systematically testing different pre-configured settings — such as learning speed, network depth, and error-penalty levels — to discover which combination produces the most accurate results on data the model has not yet seen during training.


## How it connects

This process is closely linked to gradient descent, which is the method a model uses to update its internal weights *during* training; tuning decides how fast that method moves (the learning rate). It is also tied to regularization, which is a technique for preventing overfitting, where the strength of this prevention is itself a setting you must tune. Unlike problems in 'convex optimization' — a mathematical landscape with no local traps that guarantees finding the global best solution — machine learning training surfaces are complex and non-convex (having many hills and valleys). Therefore, tuning relies on empirical trial-and-error rather than mathematical certainty.


## The mathematics

This concept draws on probability and statistics. It is not calculus or algebra in the way you might expect; instead, it is about sampling and evaluating. The mathematics you need is simple: you have a set of hyperparameters, each can take a range of values, and you want to find the combination that maximises a score (like accuracy) or minimises a loss. To do this, you evaluate many combinations. The mathematics is the act of comparing these evaluations: counting how many trials, recording the score for each, and deciding which one is best. There is no derivative, no function in the formal sense — only scores attached to trial points. Bayesian optimisation, the fanciest method, uses probability to guess which point is likely to be best next, but even that only relies on averaging and comparing, not calculus.

If H is the set of all hyperparameter combinations, and f(h) is the validation score for combination h, then tuning solves: h* = argmax_h∈H f(h). Each h is a vector like (lr, depth, reg). The grid approach enumerates all h; random search samples h uniformly; Bayesian optimisation models a probability distribution over f and chooses h that maximises expected improvement.


## A worked example

```
You have two hyperparameters: learning rate (lr) which can be 0.1, 0.01, or 0.001; and depth which can be 2, 3, or 4. That is 3 x 3 = 9 combinations. You evaluate each combination of lr and depth on a validation set and record accuracy.

Step 1: Combination 1: lr=0.1, depth=2 → accuracy 0.62   # Model trains fast but may not settle precisely.
Step 2: Combination 2: lr=0.1, depth=3 → accuracy 0.58   # Worse than before — deeper model with high lr may overshoot.
Step 3: Combination 3: lr=0.1, depth=4 → accuracy 0.45   # Even worse — overfits or diverges.
Step 4: Combination 4: lr=0.01, depth=2 → accuracy 0.71   # Slower learning helps this shallow model.
Step 5: Combination 5: lr=0.01, depth=3 → accuracy 0.78   # The sweet spot — moderate depth and moderate lr.
Step 6: Combination 6: lr=0.01, depth=4 → accuracy 0.74   # Depth adds capacity but lr still too high for best fit.
Step 7: Combination 7: lr=0.001, depth=2 → accuracy 0.69   # Slow learning underfits a shallow model.
Step 8: Combination 8: lr=0.001, depth=3 → accuracy 0.73   # Close to but not better than combination 5.
Step 9: Combination 9: lr=0.001, depth=4 → accuracy 0.75   # Deep and slow — nearly matches but not the best.

Result: The best combination is lr=0.01, depth=3 with accuracy 0.78. This is what grid search finds by trying all nine — but note it wasted four trials on lr=0.1 which never worked. Random search would have spent fewer trials on bad ranges and possibly found a good point sooner.
```


## From scratch, in Python

```python
import itertools

# We have two hyperparameters: learning_rate and depth
# We define the possible values for each
learning_rates = [0.1, 0.01, 0.001]
depths = [2, 3, 4]

# This is a list of all combinations (grid search)
combinations = list(itertools.product(learning_rates, depths))

# We will pretend we have a function that trains a model and returns accuracy
# In reality this would train a model, but for this example we will just invent scores

def fake_train(lr, depth):
    # Pretend: the best is lr=0.01, depth=3
    if lr == 0.01 and depth == 3:
        return 0.78
    elif lr == 0.01 and depth == 4:
        return 0.74
    elif lr == 0.001 and depth == 4:
        return 0.75
    else:
        return 0.6

best_score = 0
best_combo = None

for lr, depth in combinations:
    score = fake_train(lr, depth)
    print(f"lr={lr}, depth={depth} -> score={score}")
    if score > best_score:
        best_score = score
        best_combo = (lr, depth)

print("Best combination:", best_combo, "with score", best_score)
```


## The same thing, with a framework

```python
from sklearn.model_selection import ParameterGrid, train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

param_grid = {'n_estimators': [10, 50, 100], 'max_depth': [2, 3, 4]}
best_score = 0
best_params = None

for params in ParameterGrid(param_grid):
    model = RandomForestClassifier(**params)
    model.fit(X_train, y_train)
    score = model.score(X_val, y_val)
    print(params, score)
    if score > best_score:
        best_score = score
        best_params = params

print('Best:', best_params, best_score)
```


## What the code does

We start by importing `itertools`, which lets us make all combinations of two lists — that gives us the grid. Then we define the possible values for each hyperparameter. `combinations` holds every pairing: (0.1,2), (0.1,3), (0.1,4), (0.01,2), and so on. Then we define a stand-in `fake_train` function that pretends to train a model and return an accuracy score — in the real world this would be your actual model training code. We set `best_score` to 0 and `best_combo` to None. Then we loop over every combination: call `fake_train`, print the result, and if the current score is higher than the best we've seen, record it as the new best. At the end, print the winning combination. This is grid search from scratch.


## Common misconceptions

**The model learns its hyperparameters** — The model will figure out the right learning rate on its own.

Hyperparameters are set before training and never changed by the gradient descent procedure, except when you schedule them manually. In fact: Hyperparameters are fixed by you. Only the weights and biases are learned from data.

**The more hyperparameters I tune, the better** — Tuning every possible setting will always improve my model.

More knobs means more trials and a higher chance of choosing a combination that happened to work well on the validation set but not on new data. In fact: Tune only the settings that plausibly matter — and prefer random search over grid when there are many.

**Grid search is the best method** — Trying every combination systematically is the most reliable way to find the best.

When only a few dimensions matter, grid search wastes most trials on dimensions that do nothing. In fact: Random search samples more broadly, and Bayesian optimisation focuses trials where the best is likely.

**I can use the test set to choose and then report its accuracy** — The test set is the most accurate measure, so why not tune on it?

Once you have chosen hyperparameters using the test set, the test set has been 'leaked' into your decision — the reported accuracy is optimistically biased. In fact: Tune on a validation set, keep the test set untouched, and evaluate only once at the end.

**A higher learning rate always means faster training** — If the model is slow to learn, just increase the learning rate.

Above a certain threshold the loss diverges — the model jumps over the optimum and gets worse, not just faster. In fact: There is a sweet spot: too low crawls, too high diverges. Tuning finds the middle.


## In short

Hyperparameters are the knobs you turn before training — the model never learns them. Search over them with grid, random, or Bayesian methods; random beats grid when only a few knobs matter. Never tune on the test set — use validation data only. The learning rate is the most consequential knob of all.


## Where this is used

Hyperparameter tuning is the everyday work of applied machine learning. In large language models, tuning controls pretraining and fine-tuning learning rate schedules, batch size, and the number of layers — a single wrong choice can waste a multi-million-dollar training run. For small models on devices, the depth and width must be tuned to fit within memory and battery budgets while keeping accuracy. In retrieval-augmented generation, hyperparameters control the number of retrieved documents and the similarity threshold — a developer sets these by tuning on a validation set of queries. In agentic systems, tuning controls how many tool-call iterations the agent may attempt before giving up, and the temperature that controls how creative the model's responses are. In recommendation, vision, fraud detection, and speech, tuning is the most common task a data scientist does: selecting tree depth, learning rate, regularisation strength, and the number of neighbours — often automated with libraries like Optuna or scikit-learn's GridSearchCV. What breaks when tuning is done badly: overfitting the validation set — your notes record this: tuning against the test set silently invalidates the reported performance. Every domain has the same failure mode: using the test set to choose hyperparameters, then reporting test accuracy as though the model were never improved by seeing it. Also, naive grid search over too many dimensions wastes compute; random search, as your library notes emphasise, beats grid search when only a few dimensions actually matter. Practitioners write search loops in code, wrap them in Optuna trials, and often use early stopping to abort unpromising trials cheaply — a developer writing a tuning script is writing a loop that trains a model many times, each with a slightly different set of knobs.


---

Read this on the web: https://basavapurushottam.com/learn-ai/hyperparameter-tuning
