# Cross-Validation Partitions

Splitting data into folds so every observation serves once as validation, giving a less variable performance estimate than a single holdout.

**Rests on:** Hypothesis Testing, Sampling Distributions

**Leads to:** Risk Tiers

*Mathematical Foundations · Statistical Inference*


## Why learn this

You are likely to build or evaluate a real AI system. Any model — a loan predictor, a chatbot, a fraud detector — needs to know how well it will perform on data it has never seen. A single test split can be lucky or unlucky. Cross-validation gives a more reliable estimate. It is the standard practice in machine learning, used by every serious practitioner. Learning it unlocks further study: model selection, hyperparameter tuning, avoiding overfitting, and understanding evaluation reports you will encounter in industry.


## The intuition

Imagine you are a teacher with 100 students' exam scores, and you want to know the average score of all students in the school. You cannot ask everyone, so you pick a sample. But if you pick one sample, the average might be off — maybe you happened to pick the top students. Cross-validation is like repeatedly sampling different groups, computing the average each time, and then averaging those averages. It gives you a more stable and trustworthy estimate than a single sample. In machine learning, we split data into 'folds' (like groups), train on some folds, test on the remaining fold, and rotate so that each fold gets its turn being the test set. Then we average the test scores across all folds.


## An analogy

Think of a chef perfecting a recipe. She has a batch of 100 identical muffins. To taste-test, she could bake them all, then taste one muffin. That one muffin might have burnt raisins or an extra nut — not representative. So instead, she bakes 10 batches of 10 muffins each, tastes one muffin from each batch, and then takes the average taste. More work, but the final judgment is far more reliable. The same goes for cross-validation: we train on 9/10 of the data, test on the remaining 1/10, repeat 10 times (each time leaving out a different slice), and average the performance. 

But the analogy breaks down: in cooking, the muffin you taste is the same as the ones you serve. In machine learning, you never test on data you trained on — that would be cheating (like grading students on the exact homework you assigned). Cross-validation ensures the test slice is always unseen during training, giving an honest measure of generalization.


## Definition

Cross-validation is a method for estimating a model's performance by partitioning data into multiple folds, training on some folds and validating on the remaining fold, repeating this process so that every observation is used for validation exactly once, and then averaging the validation scores.


## How it connects

This concept builds on basic ideas of data splitting and model evaluation (which you may have encountered), but it assumes no prior mathematics beyond arithmetic. Cross-validation is a foundational tool in statistical inference — it helps you make conclusions about how a model will perform on new data. It connects to your notes on Model Evaluation and Overfitting (both neighbours in your library). Unlike a single holdout, cross-validation gives a less variable estimate. It is closely related to the idea of estimating generalisation error, which is central to all of machine learning.


## The mathematics

Cross-validation is a method for estimating how well a model will work on new, unseen data. It does this by repeatedly splitting the available data into two parts: a training set and a validation set.

The core mathematical operation is averaging. We assume that the performance of a model depends on the specific data it sees. By testing the model on different subsets of the data, we get multiple estimates of its performance. The average of these estimates is our final cross-validation score.

Here is how the splitting works for K-fold cross-validation:

1. Start with N total observations (data points).
2. Choose a number K (for example, 5 or 10). This is the number of folds.
3. Divide the N observations into K groups of equal size. Each group has N divided by K observations.
4. For each group k (where k goes from 1 to K):
   - Use the other K minus 1 groups as the training set. The size of this training set is N times (K minus 1) divided by K.
   - Use the single group k as the validation set. The size of this validation set is N divided by K.
   - Train a model on the training set.
   - Test that trained model on the validation set to get one performance score, let us call it score_k.

After doing this for all K groups, you have K scores: score_1, score_2, ..., score_K.

The final cross-validation score is the average of these K scores. To calculate the average, you add all the scores together and divide by K.

CV_score = (score_1 + score_2 + ... + score_K) divided by K

Where:
- CV_score is the final estimated performance.
- K is the number of folds (groups).
- score_i is the performance score obtained when group i was held out as the validation set.

Why do we average? The reference passages explain that a single split can be misleading. For example, if the validation set happens to contain easy-to-predict data points, the score will be artificially high. If it contains hard points, the score will be low. By averaging over many different splits, we reduce the influence of any single lucky or unlucky split. This gives a more stable estimate of how the model will perform on new data.

However, there is a trade-off. The reference passages note that if K is small (like 5), the training sets are smaller than the full dataset. If the model's performance improves significantly with more data, using smaller training sets can cause the cross-validation score to underestimate the true performance (this is called bias). Conversely, if K is large (like N, which is called leave-one-out), each validation set has only one point, making the estimate very noisy (high variance). A common choice like K=5 or K=10 tries to balance this.


## A worked example

Let us walk through a specific case with real numbers.

Suppose we have N = 12 data points. We want to use 3-fold cross-validation, so K = 3.

Step 1: Determine the size of each group.
Each fold has N divided by K observations.
Size of each fold = 12 divided by 3 = 4 observations.
So, we have three groups of 4 points each. Let us label the points 1 through 12.
Group 1: Points {1, 2, 3, 4}
Group 2: Points {5, 6, 7, 8}
Group 3: Points {9, 10, 11, 12}

Step 2: Perform the first fold (k=1).
- Validation set: Group 1 (Points 1-4).
- Training set: The other groups, Group 2 and Group 3. This includes Points 5 through 12.
- Train a model on Points 5-12.
- Test this model on Points 1-4.
- Suppose the accuracy is 80%. So, score_1 = 80.

Step 3: Perform the second fold (k=2).
- Validation set: Group 2 (Points 5-8).
- Training set: The other groups, Group 1 and Group 3. This includes Points 1-4 and Points 9-12.
- Train a model on Points 1-4 and 9-12.
- Test this model on Points 5-8.
- Suppose the accuracy is 75%. So, score_2 = 75.

Step 4: Perform the third fold (k=3).
- Validation set: Group 3 (Points 9-12).
- Training set: The other groups, Group 1 and Group 2. This includes Points 1-8.
- Train a model on Points 1-8.
- Test this model on Points 9-12.
- Suppose the accuracy is 85%. So, score_3 = 85.

Step 5: Calculate the average.
We add the three scores and divide by K (which is 3).
Sum of scores = 80 + 75 + 85 = 240.
CV_score = 240 divided by 3 = 80.

Result: The cross-validation accuracy is 80%.

Note on the training set size: In each step, the training set had 12 minus 4 = 8 points. This is 2/3 of the total data. As the reference passages warn, if our model needs more than 8 points to perform at its best, this estimate might be slightly lower than what we would get with all 12 points. But it is a fairer estimate than using just one split.


## From scratch, in Python

```python
# We have a list of 12 data points (just indices for simplicity)
data = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]

# Number of folds
k = 3

# Compute the size of each fold
fold_size = len(data) // k

# Create a list to store the validation scores
scores = []

# Loop over each fold
for i in range(k):
    # Determine the start and end indices of the validation fold
    start = i * fold_size
    end = start + fold_size
    
    # Validation indices are data[start:end]
    validation_indices = data[start:end]
    
    # Training indices are all other indices
    training_indices = data[:start] + data[end:]
    
    # Print what is happening (simulating training and validation)
    print("Fold", i+1)
    print("  Training:", training_indices)
    print("  Validation:", validation_indices)
    
    # Imagine we train a model on training_indices and compute accuracy on validation_indices
    # For the example, let's hardcode some hypothetical scores
    scores.append([0.80, 0.75, 0.85][i])

# Average the scores
cv_score = sum(scores) / k
print("Cross-validation score:", cv_score)
```


## The same thing, with a framework

```python
from sklearn.model_selection import KFold
import numpy as np

# Example data: 12 samples
X = np.arange(12)

# Create a 3-fold splitter
kf = KFold(n_splits=3, shuffle=False)

# Iterate over the splits
for train_index, val_index in kf.split(X):
    print("Train:", train_index, "Validation:", val_index)

# In practice you would use cross_val_score:
# from sklearn.model_selection import cross_val_score
# scores = cross_val_score(model, X, y, cv=3)
# mean_score = scores.mean()
```


## What the code does

The code creates a list of 12 data indices. We set k=3. We compute fold_size as 12 // 3 = 4. We create an empty list 'scores' to store validation results. The for loop runs 3 times (i=0,1,2). For each iteration, we compute the start and end indices for the validation part, then slice the data to get validation_indices. The training_indices are all the other data (by concatenating the list before start and after end). We print these to see the partition. We append a hypothetical accuracy score (0.80, 0.75, 0.85) for each fold. After the loop, we sum all scores and divide by k to get the average, which is 0.80.


## Common misconceptions

**More folds is always better** — If 5-fold cross-validation is good, then 100-fold must be even better.

More folds means each training set is larger, but the validation set becomes very small, so the performance estimate on each fold has high variance. Also, the computational cost increases linearly with the number of folds. In fact: The typical choice is 5 or 10 folds, balancing variance and cost. The bias-variance tradeoff means that an intermediate number of folds often gives the most reliable estimate.

**Cross-validation removes the need for a separate test set** — If I do cross-validation, I can use all my data for training and validation, and I don't need a hold-out test set.

Cross-validation is used to choose a model or tune hyperparameters, but the final model should still be evaluated on a completely unseen test set to get an unbiased estimate of performance. Cross-validation estimates performance on generalisation but the model selection process can still overfit to the specific folds. In fact: Keep a separate test set that is never touched during model selection or tuning. Use cross-validation on the training portion to choose the model, then evaluate the chosen model on the test set.

**Cross-validation is just about averaging accuracy** — It is just a fancier way to compute the average accuracy.

The key is that each data point is used for validation exactly once, which ensures the estimate is not biased by overfitting on a lucky split. Also, other metrics like precision, recall, or F1 can be averaged, not just accuracy. In fact: Cross-validation provides a more reliable estimate of a chosen performance metric by systematically rotating the validation set, reducing the variance due to a single random split.

**Cross-validation works even if preprocessing is done on the whole dataset** — I can normalize or scale all my data first, then just run cross-validation on the processed data.

If you compute the mean and standard deviation for scaling on the entire dataset, including the validation folds, then the validation folds are no longer independent — information from the validation set leaks into the training set, inflating the performance estimate. In fact: Preprocessing steps like normalisation, feature selection, or imputation must be fitted only on the training folds, and then applied to the validation fold. This is often called a 'pipeline' and is crucial to avoid data leakage.


## In short

Cross-validation partitions data into k folds, trains and validates k times, and averages the validation scores. It gives a less variable performance estimate than a single holdout. Remember: preprocess within folds, use stratification for class imbalance, and consider temporal or group-based splits when data is not independent.


## Where this is used

Cross-validation is used almost universally in machine learning. In large language models (LLMs), training data itself is often split into training and validation sets, and sometimes a k-fold style approach is used for hyperparameter tuning or for evaluating different model checkpoints. For example, when fine-tuning a smaller model on a specific task, practitioners commonly use 5-fold cross-validation to choose the learning rate and number of epochs. In small models running on devices (like keyword spotters or predictive text on a phone), cross-validation helps select a model that generalises well despite limited data. In retrieval-augmented generation (RAG), cross-validation can be used to evaluate different embedding models or retrieval strategies by splitting the query-answer pairs into folds and testing on each fold. In agentic systems that plan and call tools, cross-validation is less common because of the cost and complexity of simulating agent interactions, but simpler components (like a tool selector or a planner) can be evaluated with cross-validation on historical logs. In everyday applied ML — recommendation systems, forecasting, fraud detection, speech recognition — cross-validation is the standard way to estimate model performance before deployment. A developer typically writes `cross_val_score(model, X, y, cv=5)` in scikit-learn, which internally does exactly what we did manually: split the data into 5 folds, train on 4, test on 1, repeat, and average. When done badly — for example, preprocessing the whole dataset before splitting, or using a random split when the data has temporal structure — the cross-validation estimate becomes optimistically biased and the model may fail in production. That is why understanding the mechanics of folds matters.


---

Read this on the web: https://basavapurushottam.com/learn-ai/cross-validation-partitions
