# Continued Pre-Training

Extending pretraining on domain-specific corpora before any task tuning, to inject vocabulary and knowledge a base model lacks.

**Rests on:** Foundation Models, Pre-Trained Model Adaptation

**Leads to:** FinBERT and BloombergGPT

*LLMs and Generative AI · Fine-Tuning*


## Why learn this

Continued pre-training is the step that gives a base model the vocabulary and knowledge of a specialized field before you fine-tune it for a task. Master this and you can adapt ChatGPT-style models to medicine, law, finance, or your company's internal jargon. This unlocks: knowing when to use continued pre-training vs. plain fine-tuning, avoiding catastrophic forgetting (your model forgetting general skills), designing the right data mix, and eventually combining it with instruction tuning and LoRA to build practical AI systems that actually understand your domain.


## The intuition

Imagine you are teaching a new employee who has a university degree in general studies but knows nothing about your specific industry. They can read and write fluently, but when you say "the counterparty defaulted on the swap," they understand each word separately but not what the phrase means together. You have two options: send them to get another degree (that's pretraining a model from scratch — far too expensive), or just give them a few months of reading your company's documents, contracts, and industry reports. After reading those, they still need to learn the specific tasks you want them to do, but now they can at least understand what they're reading. Continued pre-training is exactly that second step: taking a model that was trained on the whole internet, and letting it read a large amount of domain-specific text so it learns the words, phrases, and facts of that field, before you ever teach it a specific task. The model is not learning how to answer questions or follow instructions — it is learning the raw material: the language of the domain.


## An analogy

Think of a chef who trained in French cuisine. She is brilliant, but you are opening a regional Indian restaurant, and she has never cooked with mustard oil, or used asafoetida, or seen a recipe for pakhala. You could send her to a five-year culinary school for that cuisine — but that is slow and expensive (like training a model from scratch). Instead, for two weeks, she just reads cookbooks, watches grandmothers cook, and tastes dishes. She is not learning to run a kitchen yet; she is just absorbing the flavors, ingredients, and techniques. That is continued pre-training: you are not teaching the model to answer questions or classify anything; you are giving it the raw ingredients — the vocabulary and the factual background of your domain. After those two weeks, she still needs to learn how to run your particular kitchen (that's fine-tuning), but she now understands what the recipes say. The analogy breaks, however, in one important way: the model is not a human with a body or taste buds. It only reads text, so it cannot taste or smell anything; it can only learn the patterns of words and facts that appear in the text. If the important knowledge is not in the documents, the model will never gain it.


## Definition

Continued pre-training is the process of extending a base model's training on a large, domain-specific corpus of text, before any task-specific fine-tuning, so that the model learns the vocabulary, style, and knowledge of that domain, while leaving its general conversational abilities intact as much as possible.


## How it connects

You already have notes on Large Language Models and Pretraining. This concept sits between full pretraining (training from scratch on trillions of tokens — prohibitively expensive) and fine-tuning (training on a smaller, task-specific dataset). It is also closely related to your notes on Catastrophic Forgetting and Domain Adaptation, because continued pre-training is a kind of domain adaptation, and the risk that it will forget the general knowledge is the same risk you have noted for fine-tuning. Finally, on this page you will see words like 'loss', 'weights', and 'gradient' — all of these were introduced in your notes on exactly those topics, so you can rely on that here.


## The mathematics

Continued pre-training is not a new mathematical operation, but it is a new application of the fundamental operations you already know: the dot product (which you studied when learning about neurons) and the derivative (which you studied when learning about gradient descent). This concept uses the mathematics of 'training', which is about minimizing a 'loss'. The loss is a number that tells us how wrong the model is: if the model predicts the next word in a sentence, and the real next word is different, then the loss is high. If the model predicts the same word, then the loss is low. The goal of continued pre-training is to reduce this loss, but only on the domain-specific text. To do that, we use the same gradient descent you learned when training a neuron, but on billions of parameters at once. The branch of mathematics is calculus (derivatives) and linear algebra (the dot product), but the key idea here is not a new formula, but how we use it — the schedule: when to stop, how much data to use, and how to balance it with general data.

The loss for a single prediction is calculated as follows. Let w be the weight of the model. Let x be the input value. The model's output is the product of w and x. Let y be the target value (the correct answer). The error is the difference between the output and the target. The loss is the square of that error.

Output = w times x
Error = Output minus y
Loss = Error squared

To find out how to change w, we look at the derivative of the loss with respect to w. This tells us the direction in which w must move to make the loss smaller.


## A worked example

``` 
To keep the numbers manageable, pretend the model has only one weight, w. The model's output (for one word) is output = w times 3. The true next word in the domain text would give the value 6. Loss = (output - true)^2.

Step 1: Assume the current weight w is 2. 
Calculate the output: output = 2 times 3 = 6.

Step 2: Calculate the error. 
The target value is 6. 
Error = Output minus Target = 6 minus 6 = 0.

Step 3: Calculate the loss. 
Loss = Error squared = 0 squared = 0.
This means the prediction was perfect.

Step 4: Now assume the weight w is 1. 
Calculate the output: output = 1 times 3 = 3.

Step 5: Calculate the error. 
The target value is still 6. 
Error = Output minus Target = 3 minus 6 = -3.

Step 6: Calculate the loss. 
Loss = Error squared = (-3) squared = 9.
This means the prediction was wrong, and the loss is 9.

Result: The loss tells us how far off we are. Training uses derivatives to adjust the weight so that the loss becomes small.
```


## From scratch, in Python

```python
# Continued pre-training with a tiny toy model, in plain Python
# We have a corpus of domain words. We make a simple bigram model:
# it predicts the next word given the previous word.
# No frameworks, just dictionaries and loops.

text = "the cat sat the dog ran the cat slept"
words = text.split()

# Build a dictionary: key = word, value = list of words that came next
bigrams = {}
for i in range(len(words)-1):
    prev = words[i]
    nxt = words[i+1]
    if prev not in bigrams:
        bigrams[prev] = []
    bigrams[prev].append(nxt)

# 'the' -> ['cat','dog','cat']

# Simple function to predict next word for a given previous word
def predict(prev):
    if prev not in bigrams:
        return None
    # choose the most common next word
    from collections import Counter
    counts = Counter(bigrams[prev])
    # most_common(1) returns a list of (word, count) tuples
    return counts.most_common(1)[0][0]

print(predict('the'))  # most common next word after 'the'

# Now, to demonstrate 'continued pre-training', we start with a model
# that was trained on general text, then we will 'continue' it on
# domain text (here the same text, but we simulate it).
# In real life, we would have two separate corpora.

# For this demo, we'll just show the model's parameters (here the bigram counts)
# and imagine we add more domain sentences to update them.

# Here is how you would 'train' a simple count-based model: add more data
# and rebuild the counts.
more_domain_text = "the cat ran the dog slept"
all_text = text + " " + more_domain_text
all_words = all_text.split()
bigrams.update? # not needed, we just rebuild
# Actually rebuild:
bigrams = {}
for i in range(len(all_words)-1):
    prev = all_words[i]
    nxt = all_words[i+1]
    if prev not in bigrams:
        bigrams[prev] = []
    bigrams[prev].append(nxt)

print(predict('the'))  # Now after more domain data, 'the' -> still 'cat' but counts changed

```


## The same thing, with a framework

```python
null
```


## What the code does

The code builds a very small 'model' from a text corpus. A dictionary maps each word (key) to a list of words that followed it (value). For example, in the text 'the cat sat the dog ran the cat slept', the word 'the' appears four times, and the next words are 'cat', 'dog', 'cat', so the dictionary for 'the' is ['cat', 'dog', 'cat']. The predict function finds the most common next word using collections.Counter, which counts how many times each word appears in a list. Then we simulate continued pre-training by adding more domain text and rebuilding the counts. In real models, the 'weights' are matrices, and training is done via gradient descent on billions of numbers, but the principle is the same: you show it domain text, and you adjust its internal numbers to make it more likely to produce the correct next word.


## Common misconceptions

**Continued pre-training is the same as fine-tuning** — Fine-tuning is fine-tuning, continued pre-training sounds like just another name.

Fine-tuning is task-specific: it adjusts the model to answer questions or classify text. Continued pre-training is task-agnostic: it only teaches the model domain language and knowledge, with no specific task in mind. In fact: Continued pre-training happens before fine-tuning and uses raw text to teach the model the domain. Fine-tuning uses labelled examples to teach a task.

**More domain data is always better** — If I just feed in all my company's documents, continued pre-training will make the model perfect.

Feeding too much repetitive or low-quality data can cause the model to overfit to that data and forget general knowledge (catastrophic forgetting). In fact: Data quality, diversity, and mixing in general data matter more than raw volume. You need to balance domain data with general text to retain general ability.

**It will teach the model facts perfectly** — If the model reads all the textbooks, it will know all the facts in them.

The model does not have a memory; it learns patterns. It can memorize some facts, but it may also hallucinate or get facts wrong. In fact: It learns the vocabulary and statistical associations, not a reliable database. For reliable facts, use retrieval-augmented generation rather than relying on the model's internal memory.

**It is only for large companies** — Only big firms with huge budgets can do continued pre-training.

Now open-weight models and affordable hardware make it possible for small teams and even individual developers, especially with parameter-efficient methods like LoRA. In fact: Open models like Llama and Qwen can be continued pre-trained on a single GPU with the right libraries and quantization.


## In short

Continued pre-training teaches a general model the vocabulary and facts of a domain by training it further on domain-specific text, before any task tuning. It is distinct from fine-tuning, which teaches a specific task. The key is to use a low learning rate and mix in some general data to prevent catastrophic forgetting. It is a necessity when the domain's language is unfamiliar to the base model, and it is the bridge between a generic pretrained model and a model that truly understands your field.


## Where this is used

In practice, continued pre-training is used across AI development:

* **Large language models (LLMs):** Companies like OpenAI and Google do not publicly continue pre-training on their biggest models, but organizations that want to specialize a model — e.g., for legal or medical text — will take an open-weight model like Llama or Mistral and continue training it on millions of legal cases or medical journals. This gives the model the domain vocabulary and knowledge. For example, a model that has never seen the acronym 'CPT' (Current Procedural Terminology) will learn that it refers to medical billing codes. The mechanism is the same loss function you know: predict the next token on domain text, and repeat billions of times.
* **Small models on devices:** When running on a phone or laptop, you want a small model (like a 1B parameter model) that still understands domain language. Continued pre-training a small model on domain text can make it as good as a larger model for that domain, while keeping it fast enough to run locally. For instance, a smart keyboard's autocomplete might be continued pre-trained on the user's own writing style.
* **Retrieval-augmented generation (RAG):** RAG systems retrieve relevant documents and feed them to the model. If the model does not understand the domain's jargon, the retrieved text is useless. Continued pre-training on domain documents makes the model understand the retrieved text better, improving the quality of the generated answers. The embedding model used for retrieval also benefits from continued pre-training on domain text, so that similar documents are mapped closer in vector space.
* **Agentic systems:** Agents that plan and call tools (e.g., a financial analysis agent that queries a stock database) need to understand user requests like 'show me the P/E ratio of Tata Motors'. Continued pre-training on financial text teaches the model what 'P/E' means, so it can correctly plan to call the right API. Without that, it might misinterpret the request.
* **Everyday applied ML:** Recommendation systems, fraud detection, and speech recognition all use text understanding models. For example, a bank's fraud detection model reads transaction descriptions like 'UPI transfer to XYZ' — continued pre-training on such transaction descriptions teaches the model the specific vocabulary, improving fraud detection accuracy.

Developers use libraries like Hugging Face Transformers and PyTorch. They load a pretrained model, load their domain corpus (e.g., a text file), tokenize it, and run training loops with a language modeling loss (predict next token). They use a low learning rate (like 1e-5) to avoid destroying the base knowledge. They often mix in some general text to prevent catastrophic forgetting. If done badly — too little data, too high a learning rate, or forgetting to mix general data — the model may lose common sense or use only the narrow domain, and benchmarks on general tasks will drop.


---

Read this on the web: https://basavapurushottam.com/learn-ai/continued-pre-training
