# Synthetic Data Generation

Using models to produce training data — instructions, preferences, self-play trajectories — where human annotation is too slow or costly.

**Rests on:** Instruction Tuning, Pre-Trained Model Adaptation

**Leads to:** Continuous Learning Loops, On-Policy vs Off-Policy, Preference Data Collection

*LLMs and Generative AI · Post-Training Alignment*


## Why learn this

This concept explains how modern AI systems learn from vast amounts of practice material without requiring humans to write every single example by hand. As you will see when studying further alignment methods, every one of these methods is only as good as the data they are trained on. Synthetic data generation is how current AI produces that data at scale when human annotators are too slow or too expensive. Understanding this first means you will recognise what is really happening when those advanced methods talk about comparing a 'good' answer against a 'bad' one, or when they describe a model learning by playing against its own previous versions. For example, consider the specific step of generating 10,000 practice questions for a math tutor AI; doing this by hand is impossible, but synthetic generation makes it routine.


## The intuition

Imagine you are a teacher grading essays. You have one perfect student whose work you trust implicitly. You need more essays to grade to ensure the class is improving, but you only have a handful of real ones available. So you ask that trusted student to write many more essays on similar topics. The new essays are not written by you directly, but they still carry the style, tone, and quality of the original trusted work. Now you have hundreds of essays to review, instead of just a dozen. Synthetic data generation is exactly this process: using an AI model (the 'perfect student') to produce new pieces of training data that resemble human-created data. The key word here is 'produce' — the model is creating new examples from scratch, not merely copying or storing old ones. In AI, you often need thousands of examples to train a model properly, but collecting that many high-quality human-written examples is slow and expensive. So instead, you use a model to generate them. The catch, as with the student, is that the new data inherits the model's own habits and biases — you are not getting fresh unbiased human perspective; you are getting a refined copy of the model's own view. If the model initially struggled with sarcasm, it might now generate 500 sarcastic examples that all miss the nuance, reinforcing that specific weakness rather than correcting it.


## An analogy

Think of a chef training an apprentice to cook. A good apprentice needs to taste many dishes to learn what good food is. The chef could cook every dish by hand, but that is exhausting and slow. So after the chef has cooked a few excellent examples, the apprentice copies those and invents variations. Those variations are synthetic training data. At first the apprentice produces dishes that are close to the chef's high standard, but gradually the apprentice introduces personal quirks — perhaps too much salt or a strong preference for cilantro. This is the same amplification of bias that happens when a model generates its own training data; small errors get repeated and reinforced over time. That said, the analogy stops working at one point: in advanced learning loops, the apprentice is also the one cooking the food to study it — like a chess player who only plays against their own past self. This can still improve skill through repetition, but it can also reinforce blind spots because no external correction is coming in.


## Definition

Synthetic data generation is the process of using an AI model to automatically create training examples—such as instructions, preference comparisons, or reasoning steps—from scratch, rather than collecting them manually from human annotators.


## How it connects

This concept sits directly under 'Post-Training Alignment' (the phase where an AI adjusts its behavior to match human values after its initial general knowledge training) in your library. It supports several neighbouring topics: Preference Data Collection, which normally requires human annotators but can be accelerated by using models to generate comparison pairs; techniques like Direct Preference Optimization and GRPO, which both rely on comparing a 'good' response against a 'bad' one that are increasingly generated by the model itself; and Reward Modeling (the process of teaching an AI how to score or judge the quality of answers), where the system that judges quality is often trained on synthetic comparisons produced by the main model. Understanding synthetic data generation helps you see why experts say 'preference data quality dominates algorithm choice' — because the quality of that data is often determined by how it was generated and whether it was filtered for errors.


## The mathematics

The mathematics here is not about complex formulas — it is about counting and understanding probability. The core operation is simple: when a model generates text, it does not pick words at random. It assigns a probability to every possible next word, given all the words before it. For example, after 'The sky is', the word 'blue' might have a high probability, and 'banana' a very low one.

Synthetic data generation exploits this by sampling from those probabilities. You 'turn up the randomness' to get varied outputs, or 'turn it down' to get more predictable ones. The branch of mathematics involved is probability — because we are working with chances and distributions, not certainties. The specific operation is sampling: choosing a word according to its probability, then moving to the next word, and so on, until a full sentence is produced.

Even though we do not write equations for it, every generated sentence is the result of many tiny probabilistic choices.

No single formula is needed for this concept. The only mathematical idea is sampling from a probability distribution. If the model gives probabilities p1, p2, ..., pn for the next word, you pick a random number r between 0 and 1, and choose the word where the cumulative probability first reaches or exceeds r. That is it.

To understand why this works, consider how we define 'probability' in this context. Probability is a number between 0 and 1 that tells us how likely an event is. If an event has probability 0, it never happens. If it has probability 1, it always happens. If it has probability 0.5, it happens half the time in the long run.

When we say the model 'assigns a probability' to each word, we mean it has calculated how often that word appears after the current context in its training data. For instance, if the phrase 'The sky is' appeared 100 times in the training data, and 'blue' followed it 80 times, the model assigns a probability of 0.8 to 'blue'.

Sampling is the process of using these probabilities to make a choice. Imagine you have a bag with colored balls. If 80% of the balls are blue and 20% are red, and you pick one ball at random, you are more likely to get a blue ball. This is exactly what the model does. It creates a virtual 'bag' of words where each word appears a number of times proportional to its probability. Then it picks one word at random.

This process is repeated for every word in the sentence. After picking the first word, the context changes. The new context includes the word just picked. The model then recalculates the probabilities for the next word based on this new context. This is called conditional probability: the probability of an event given that another event has already happened.

For example, if the sentence so far is 'The cat sat', the next word might be 'on' with high probability. But if the sentence was 'The cat jumped', the next word might be 'over' with high probability. The model uses the entire history of words to determine the probabilities for the next step.

This is why synthetic data generation is powerful. By controlling the probabilities (or 'temperature'), we can control the diversity of the generated text. If we make the probabilities more equal, the model picks words more randomly, leading to more diverse but potentially less coherent text. If we make the probabilities more extreme (some very high, some very low), the model picks the most likely words, leading to more predictable and coherent text.

The key mathematical concept here is that each step in the generation process is independent of the others, given the current context. This means the choice of the next word depends only on the words before it, not on words after it. This is a fundamental assumption in many language models.

In summary, synthetic data generation uses probability theory to create new data. It does this by repeatedly sampling from conditional probability distributions. Each sample is a new piece of data that follows the patterns learned from the training data.


## A worked example

``` 
We have a tiny language model that must generate the next word after 'The cat'. It assigns these probabilities to three possible words: 'sat' has 0.5, 'slept' has 0.3, 'jumped' has 0.2.

Step 1: List of probabilities: sat=0.5, slept=0.3, jumped=0.2   # These sum to 1.0 — they are the model's belief about what word comes next.
Step 2: Random number drawn: 0.67   # We pick a random number between 0 and 1 to decide which word to choose.
Step 3: 0.67 falls between 0.5 and 0.8 (0.5+0.3)   # We accumulate the probabilities: 0–0.5 for 'sat', 0.5–0.8 for 'slept', 0.8–1.0 for 'jumped'.
Step 4: So we choose 'slept'   # The random number landed in the 'slept' interval.
Step 5: Now the sentence is 'The cat slept'   # We then repeat the process for the next word.

Result: The output is 'slept'. This is one sample from the model's probability distribution. By repeating this many times, we can generate many different sentences about cats.
``` 

Let us break down Step 3 in more detail. Why do we add the probabilities?

Imagine a number line from 0 to 1. We divide this line into segments, where the length of each segment is equal to the probability of the corresponding word.

- The first segment, from 0 to 0.5, represents the word 'sat'. Its length is 0.5.
- The second segment, from 0.5 to 0.8, represents the word 'slept'. Its length is 0.3 (because 0.8 - 0.5 = 0.3).
- The third segment, from 0.8 to 1.0, represents the word 'jumped'. Its length is 0.2 (because 1.0 - 0.8 = 0.2).

When we draw a random number r between 0 and 1, we are essentially picking a point on this number line. The word whose segment contains that point is the one we choose.

In our example, r = 0.67. This point lies in the second segment (from 0.5 to 0.8). Therefore, we choose 'slept'.

This method ensures that words with higher probabilities have larger segments, and thus a higher chance of being chosen. It is a fair and mathematically sound way to sample from a distribution.

Now, let us see what happens if we generate another sentence. Suppose the next word after 'The cat slept' has these probabilities:
- 'on' = 0.6
- 'in' = 0.3
- 'under' = 0.1

We draw a new random number, say r = 0.45.

- The segment for 'on' is from 0 to 0.6.
- The segment for 'in' is from 0.6 to 0.9.
- The segment for 'under' is from 0.9 to 1.0.

The point 0.45 lies in the first segment (from 0 to 0.6). Therefore, we choose 'on'.

The sentence is now 'The cat slept on'.

We repeat this process until the sentence is complete or a stopping condition is met. Each step is independent, given the current context. This is how synthetic data is generated: by chaining together these probabilistic choices.


## From scratch, in Python

```python
# Let's simulate the tiny language model from the maths section.
# We have a list of words and their probabilities.
# In Python, a list is an ordered collection - we can hold words and numbers in it.

words = ['sat', 'slept', 'jumped']
probabilities = [0.5, 0.3, 0.2]

# We need to pick a word according to these probabilities.
# We use a random number. The random module gives us one.
import random

r = random.random()  # a number between 0 and 1

# We walk through the words, accumulating the probability.
# So loop means 'repeat for each word'.
cumulative = 0
chosen_word = words[-1]  # default: last word
for word, prob in zip(words, probabilities):
    cumulative += prob
    if r <= cumulative:
        chosen_word = word
        break

print('Random number was:', r)
print('Chosen word:', chosen_word)

# Now we generate a whole sentence of length 4, using the same idea.
# But real models have thousands of words, not just three.
# Here we keep it simple.

sentence = ['The', 'cat']
for _ in range(4):  # repeat 4 times
    # Pretend this is a model outputting probabilities for each possible next word.
    # We have the same three words and the same probabilities for simplicity.
    r = random.random()
    cumulative = 0
    next_word = words[-1]
    for word, prob in zip(words, probabilities):
        cumulative += prob
        if r <= cumulative:
            next_word = word
            break
    sentence.append(next_word)

print('Generated sentence:', ' '.join(sentence))
```


## The same thing, with a framework

```python
import torch
import torch.nn.functional as F

# A real model is a neural network. For illustration, we use a random vector of logits.
# Logits are raw scores that we convert to probabilities.
logits = torch.tensor([0.5, -0.7, 1.2])  # one score per possible word

# Convert logits to probabilities: softmax does this.
probs = F.softmax(logits, dim=0)
print('Probabilities:', probs)

# Now sample one word index according to these probabilities.
# PyTorch's multinomial does exactly the sampling loop we wrote by hand.
word_index = torch.multinomial(probs, 1).item()
words = ['sat', 'slept', 'jumped']
print('Chosen word:', words[word_index])
```


## What the code does

The from_scratch code teaches two Python ideas: a list (an ordered collection that can hold words or numbers) and a for loop (repeats an action for each item in a list, or a fixed number of times). The code walks through the list of probabilities, adding each one to a running total, and stops when the random number is less than that total. That is the cumulative probability sampling from the maths section. The framework version shows the same idea in PyTorch: logits become probabilities through 'softmax', and torch.multinomial performs the same sampling internally. The learner should see that the library version is not magic — it is doing the same accumulate-and-compare loop underneath.


## Common misconceptions

**Synthetic data is free and unlimited** — Since a model generates it, I can produce as much as I want without any cost or risk.

Every generated example costs GPU time and API calls, and more importantly, low-quality synthetic data can actively harm the model's performance, so you must spend effort filtering. In fact: Synthetic data is cheaper than human annotation but not free — it requires compute, careful filtering, and quality control to be useful.

**Synthetic data is a perfect copy of real data** — The model generates examples that are just as good as human-written ones, so the training is equivalent.

The model can only reproduce what it learned from its training data; it cannot introduce new perspectives or correct its own blind spots. In fact: Synthetic data is a mirror of the generator's distribution, not an independent source of truth.

**Filtering is optional if the generator is good** — If I use a strong model like GPT-4 to generate data, I can skip the filtering step.

Even strong models produce repetitions, hallucinations, and subtle biases that degrade downstream training. Filtering is what keeps quality from collapsing. In fact: Filtering and verification are essential, regardless of the generator's apparent quality.

**Self-play is just for games** — Self-play only works in computer chess or Go, not for language models.

Language models use self-play to generate reasoning traces for math problems or code, comparing multiple solutions to find better ones. In fact: Self-play is a general mechanism where a model generates its own training trajectories, applicable to any task with a computable reward.

**Synthetic data eliminates the need for real data** — Once I have a good generator, I never need to collect human data again.

A model trained purely on synthetic data will drift further from human preferences over time, a problem called 'model collapse'. Real data anchors the process. In fact: Synthetic data augments but does not replace human annotation; a small amount of real data is often necessary to keep quality stable.

**Synthetic data is always worse than real data** — Anything generated by a machine is lower quality than anything written by a human.

For repetitive, well-specified tasks (like generating SQL queries from a schema), synthetic data can be more consistent and cheaper, and can even improve coverage. In fact: Synthetic data can be equal or better in controlled settings, but it carries the generator's biases and needs verification.


## In short

Synthetic data generation is using a model to produce training data because human annotation is too slow or costly. The core idea is sampling from a probability distribution. The critical lesson is that synthetic data scales volumes but amplifies the generator's biases — filtering and verification are what keep quality from collapsing. This single concept underlies Preference Data Collection, DPO, GRPO, and Reward Modeling, as the engine that creates the pairs and trajectories those methods need.


## Where this is used

Synthetic data generation is everywhere in modern AI, and the mechanism is always the same: a model produces text or other data that is then used to train another model, or sometimes itself.

**Large language models (LLMs).** The biggest use is in post-training. After pretraining, models are fine-tuned on instruction data — pairs of a prompt and a good response. Collecting tens of thousands of these by hand is expensive and slow. Instead, companies use a strong model (often the previous version) to generate instructions and responses, then filter them by quality. For example, in supervised fine-tuning (SFT), a model like GPT-3.5 might generate millions of instruction-response pairs, and a smaller set of human-written examples is used to calibrate quality. The mechanism: the generator model is given a topic, and it produces both the instruction and the response. Filtering then removes low-quality or repetitive outputs. In your library's terms, this is where 'demonstration quality dominates quantity' — you can generate a million, but only the filtered few thousand are useful.

**Small language models (on-device).** When a model must run on a phone or laptop, it is typically a smaller model that cannot learn complex skills from scratch. To teach it, developers often use a larger model to generate training data — this is called 'distillation'. For example, a small model for keyword spotting is trained on thousands of synthetic audio clips generated by a text-to-speech model, because recording humans in every accent is impossible. The mechanism is knowledge transfer through synthetic data: the big model's outputs become the small model's training labels.

**Retrieval-augmented generation (RAG).** In RAG, a system retrieves relevant documents and feeds them to the model to answer a question. Synthetic data is used to create training examples for the retriever — the part that decides which documents to fetch. A model generates question–answer pairs from a set of documents, then the answer is matched back to the document that contains it, creating a positive example. The mechanism is the model reads a passage and produces a likely question, giving you a natural query-document pair without manual annotation.

**Agentic systems.** Agents that plan and call tools are trained on trajectories — sequences of actions and observations. Writing these by hand is extremely tedious. Self-play generates them: the model interacts with a simulated environment (like a web API) and the whole interaction is recorded as a training example. For example, an agent learns to book flights by generating many synthetic booking attempts, some successful, some not. The mechanism is reinforcement learning: the agent's actions are the data, and the reward (did it book?) is computed automatically. This is the 'self-play trajectories' from your library notes.

**Everyday applied ML.** Recommendation systems generate synthetic user interactions to augment sparse real data. Fraud detection uses synthetic fraudulent transactions (created by modifying real ones) to train a classifier when true fraud cases are rare. Vision models use synthetic images from 3D renders to teach object recognition. The mechanism is always the same: create plausible examples algorithmically, then train on them, often mixing with real data.

**What a developer writes.** In practice, a developer might write a script that calls an LLM API with a prompt like 'Generate a customer support email and the ideal agent response', then filters the outputs by checking whether they contain forbidden words or duplicate text. They might use a library like PyTorch to sample from the model's logits, as in the framework code. What breaks when done badly: if you do not filter, the synthetic data can be repetitive, toxic, or reflect the generating model's biases, and the final model will inherit those flaws. That is the 'amplification of bias' from your notes.


---

Read this on the web: https://basavapurushottam.com/learn-ai/synthetic-data-generation
