# Foundation Models

Large models pretrained broadly enough to adapt to many downstream tasks, serving as a shared base rather than a task-specific artifact.

**Rests on:** Feed-Forward Blocks, Multi-Layer Perceptrons

**Leads to:** Continued Pre-Training, LoRA and QLoRA, Multi-Tenant Adapter Serving, Pre-Trained Model Adaptation, QLoRA for Finance, Reasoning Models, Routing and Expert Specialization

*LLMs and Generative AI · Large Language Models*


## Why learn this

You likely use tools powered by foundation models every day without realizing what they are. When you ask a chatbot to draft an email, summarize a long article, or translate a sentence in an app on your phone, you are interacting with a foundation model. Understanding this concept explains why one single system can perform hundreds of different jobs, why companies focus on 'adapting' these systems rather than building new ones for every task, and why the same tool might be incredibly helpful yet occasionally biased or incorrect. This page is the essential starting point; without grasping what a foundation model is, you will not understand how prompt engineering works, nor how AI agents operate.


## The intuition

Before foundation models became common, software was built like a specialized factory. A spam filter was trained only on spam and could only filter spam. A face-recognition app was trained only on photos of faces and could only recognize faces. Each project was separate and expensive. A foundation model changes this approach. Instead of building many single-purpose tools, engineers train one enormous model on a vast mixture of data from the internet—text, images, code, and audio. During this training, the model learns general patterns about how language and images work. Once trained, you can use that same model for thousands of different tasks: answering questions, writing emails, identifying objects, summarizing documents, or translating languages. You do not need to build a new machine for each job; you simply 'prompt' the foundation model with instructions on what you want. Think of it as the difference between a factory that makes only toasters and a skilled human worker who can assemble a toaster, build a chair, or bake a cake depending on your specific request. The foundation model is the skilled worker. You still need to tell it exactly what you want, but you do not have to hire a new expert for every single task.


## An analogy

Think of a foundation model as a general practitioner doctor. You go to the same doctor whether you have a fever, a broken arm, or a skin rash. The doctor did not attend separate, specialized schools for each condition; they attended medical school to learn the fundamentals of the human body and physiology. Now, when you present any symptom, they apply that broad foundational knowledge to diagnose and treat you. A foundation model is similar: it 'went to school' on a massive collection of text and images, learning general structures of language and vision. When you ask it a question or show it an image, it uses that broad base to respond. The crucial limit of this analogy is understanding how the model actually works. A doctor has a conscious mind that reasons and understands causality. A foundation model does not. It is a giant pattern-matching engine. It has never 'understood' anything; it has simply learned to predict what the next word or image pixel should be, based on statistical patterns in its training data. When you ask a question, it is not recalling a fact from memory but generating a sequence of text that statistically matches the pattern of a good answer. This imitation can be brilliantly useful, but it also means the model can confidently state a falsehood if the pattern suggests it should. Just as a doctor’s specific training influences their empathy and approach, the data used to train the foundation model shapes everything it does—including any biases it may have learned.


## Definition

A foundation model is a large machine-learning system trained on a broad and diverse set of data, designed so that the same core system can be adapted to perform many different jobs through instructions or slight adjustments, rather than being built from scratch for just one specific purpose.


## How it connects

You are at the beginning of your learning path here, but this concept connects directly to two key ideas you will encounter next. First, Large Language Models are the specific type of foundation model that deals with text; understanding what a foundation model is helps you see how a text-based system can mimic human conversation. Second, you need to understand how these models manage information: they process input by converting words into numerical codes (a method often called byte-pair encoding) so the math operations work correctly, and they are limited by their 'context window'—the fixed amount of prior text they can hold in memory at once for any single response. These two mechanisms determine how much context the model can use and how it represents that context internally.


## The mathematics

This concept rests on **arithmetic**, not advanced algebra. You do not need calculus or complex formulas to understand what a foundation model is. The core idea is about **scale and sharing**. 

When we talk about the model's 'size' — for example, 70 billion parameters — we mean it has 70 billion numbers inside it that get multiplied and added together every time it makes a prediction. All that arithmetic happens inside the computer, but you never have to do it yourself. The math that matters for you is simply this:

total work = (amount of data) × (amount of arithmetic per data point) × (number of training cycles)

The model does that work once during its initial training. After that, you can reuse the result for many different tasks. That is the key arithmetic insight: one big training run is cheaper than many small ones, if you can share the result. A foundation model is a shared base, so you do not need to pay for training a new model from scratch for every single task.

There is no new operation to learn on this page — it is all addition and multiplication at scale.

No new formula is introduced on this page. The only 'equation' is a cost comparison. Imagine you have T tasks. If you train each one separately, each costs C. The total cost is T multiplied by C.

With a foundation model approach, you pay P to pretrain the big general model first. Then, for each of the T tasks, you pay F to fine-tune that existing model. Your total cost is P plus T multiplied by F.

The foundation model wins when this total is less than the separate training cost:
P + (T × F) < (T × C)

This inequality tells us exactly when it makes financial sense to use a foundation model.


## A worked example

Suppose you want to train a system to do three tasks: classify emails as spam, translate English to Hindi, and summarize articles. Without a foundation model, you would train three separate models, each on its own data. Let us say each training run costs ₹1 crore (approximately $120,000). That is ₹3 crore total.

With a foundation model approach, you train one large model on general text data for ₹2 crore. It is bigger, so it costs more than one task-specific model. Then you 'fine-tune' it for each task, at a cost of ₹10 lakh per task.

Step 1: Option A: three task-specific models
You calculate the cost by multiplying the number of tasks by the cost per task:
3 × ₹1 crore = ₹3 crore
Each task needs its own model and its own training run.

Step 2: Option B: one foundation model pretraining cost
You pay a single upfront fee for the general model:
₹2 crore
You train a general model on a huge diverse dataset (text, maybe images).

Step 3: Fine-tune for each task
You add the fine-tuning costs for all three tasks. Since there are three tasks and each costs ₹10 lakh:
₹10 lakh + ₹10 lakh + ₹10 lakh = ₹30 lakh
Each task only adjusts the model slightly, which requires far less data and compute than training from scratch.

Step 4: Total cost for Option B
Add the pretraining cost to the fine-tuning costs:
₹2 crore + ₹30 lakh = ₹2.3 crore
(Note: 1 crore equals 100 lakh, so ₹2 crore is ₹200 lakh. ₹200 lakh + ₹30 lakh = ₹230 lakh, which is ₹2.3 crore).

Step 5: Savings
Subtract the total cost of Option B from the total cost of Option A:
₹3 crore − ₹2.3 crore = ₹70 lakh
The foundation model approach is cheaper when you have multiple tasks.

Result: The foundation model approach saves ₹70 lakh in this tiny example. Real-world savings are far larger, because a model like GPT-4 is used by millions of people for tens of thousands of different tasks, and users never pay for training — only for each use. The math of sharing beats the math of duplication.


## From scratch, in Python

```python
This concept is about the philosophy and economics of a foundation model, not about a specific algorithm. So there isn't a single line of code to write that demonstrates it exactly. Instead, we'll make the idea concrete with a small example: we'll build a 'toy' foundation model for a tiny dataset — a mini language model that predicts the next letter in a sequence. This is exactly what large models do, but with letters instead of words and millions of times smaller.

# Python for a toy foundation model:

# A list of training examples
sequences = ["aab", "aba", "abb"]

# Count how often each letter follows each other letter
# We'll use a dictionary: key is the current letter, value is a dictionary of next-letter counts
counts = {}
for seq in sequences:
    for i in range(len(seq)-1):
        current = seq[i]
        next_letter = seq[i+1]
        if current not in counts:
            counts[current] = {}
        # this line adds 1 to the count for (current, next_letter)
        counts[current][next_letter] = counts[current].get(next_letter, 0) + 1

# Convert counts to probabilities
import math

def predict_next(current):
    if current not in counts:
        return None
    # We have the total count for this current letter
    total = sum(counts[current].values())
    best_letter = None
    best_prob = 0
    for letter, count in counts[current].items():
        prob = count / total
        print(f"P({letter} | {current}) = {prob:.2f}")
        if prob > best_prob:
            best_prob = prob
            best_letter = letter
    return best_letter

# Now let's prompt our 'foundation model' with the start 'a'
print("Prompt: 'a'")
next_letter = predict_next('a')
print(f"Predicted next: {next_letter}")

# Let's see what happens with a different prompt 'b'
print("Prompt: 'b'")
next_letter = predict_next('b')
print(f"Predicted next: {next_letter}")

# This toy model is a 'foundation' because it can be used for ANY sequence that follows the same pattern
```


## The same thing, with a framework

```python
null
```


## What the code does

We create a list `sequences` to hold our tiny training corpus. Then we build a dictionary `counts` that records, for every letter we see, what letters come after it. The loops iterate over each sequence and each position within it. After we have the counts, we write a function `predict_next` that, given a current letter, finds which letter is most likely to follow. It does this by turning the raw counts into probabilities: count for a candidate divided by total count for that current letter. Finally we call `predict_next` with 'a' and then with 'b' to show that the same model can do a 'task' of predicting a next letter. It's not a real foundation model, but it captures the essence: one model trained on data can be used for many different prompts.

**Teaching the Python building blocks:**
- A **list** is an ordered collection that can be changed. Here `sequences = ["aab", "aba", "abb"]` — it holds three strings.
- A **dictionary** (dict) stores key-value pairs. In `counts = {}`, the key is the current letter, and the value is another dictionary of next-letter counts. For example, after processing, `counts['a']` might be `{'a': 2, 'b': 1}`.
- A **loop** (`for i in range(...)`) repeats code a fixed number of times. `range(len(seq)-1)` gives positions 0, 1, 2... up to the second-to-last, because we always look at the current and next letter.
- A **function** is a reusable block of code. `predict_next(current)` takes an input, does some computations, and returns a value. We use `return` to output the best letter.
- The line `counts[current][next_letter] = counts[current].get(next_letter, 0) + 1` says: take the inner dictionary for `current`, get the current count for `next_letter` (or 0 if not present), add 1, and store it back.
- The `sum` function adds up all values in a dictionary's values.

This is plain Python with no libraries — it's the arithmetic under the hood.


## Common misconceptions

**Foundation models are new, task-specific AI** — A foundation model is like a spam filter, but for talking — it's just a very good chatbot

A spam filter is trained for a single task: classify email as spam/not spam. A foundation model is trained to predict the next token on a huge, varied dataset. It can do thousands of things — write a poem, translate, summarize — not just one. It's general, not task-specific. In fact: A foundation model is a general-purpose machine trained on broad data, adapted to many tasks, not a single-purpose tool.

**Foundation models are omniscient** — It has read the entire internet, so it knows everything, including my private data.

It only knows what was in its training data, which is a snapshot in time and is public. It has no access to your private files. In fact: Foundation models know only their training data, which is public and finite. Private data isn't included unless explicitly included.

**Foundation model is a database of facts** — When I ask it a factual question, it looks up the answer in its memory.

The model doesn't 'look up' anything. It predicts the next word based on patterns. It can hallucinate a confident but false answer. In fact: A foundation model generates text based on patterns, not by retrieving facts. It can be wrong, so you should verify important info.

**Foundation model = one specific product** — ChatGPT is a foundation model; there's nothing else.

ChatGPT is built on a foundation model (GPT-4), but many different foundation models exist (Claude, Gemini, Llama). And they extend beyond text. In fact: Foundation models are a category of models; ChatGPT is one product built on one example.

**Fine-tuning means retraining from scratch** — To make a foundation model good at my task, I must train it all over again on my data.

Fine-tuning is a much smaller step. You start from the already-trained model and do a small number of additional training steps on your data. It takes a fraction of the time, data, and compute. In fact: Fine-tuning adapts an existing foundation model with a small additional training run — not full retraining.


## In short

A foundation model is a large model trained on a broad dataset to be adaptable to many tasks. The key insight is economic: one big model shared across many uses costs less than many task-specific models. It has no real understanding — it generates responses by predicting patterns in data. It can hallucinate, it has biases from its training data, and you must build guardrails around it. But it's the reason you can use AI without training your own. Remember: one model, many tasks, but always with limitations.


## Where this is used

In real life, foundation models are everywhere. Here are the concrete ways they're used, and what a developer actually does.

**Large Language Models (LLMs).** The best-known foundation models are LLMs like GPT-4, Claude, Gemini. They are trained on enormous text corpora to predict the next token. Developers don't retrain them; they use an API. For example, to answer a question, you send a prompt to the API, and the model returns a string. You might write: `client.chat.completions.create(model="gpt-4", messages=[{"role": "user", "content": "Rơanada's capital?"}])`. That's it. The foundation model does the work. The developer doesn't build a model that knows geography — they use a model that has learned geography from the internet.

**Small Language Models (on phones/laptops).** For your phone, you can't send data to a cloud server. You need a model that runs on-device. Here, foundation models come as 'small language models' like Phi-2 or Gemma-2, which are distilled from larger foundation models. A developer downloads a lightweight model file (hundreds of MB) and integrates it into an app. The model is still a foundation model for its size — it can do multiple tasks like autocomplete, summarization, and simple Q&A. The tradeoff is quality vs. resource use: the smaller model runs fast and privately, but it's less capable than the huge one. The mechanism is the same, just scaled down.

**Retrieval-Augmented Generation (RAG).** When you want a foundation model to use your own documents (say, your company's manual), you can't retrain it. Instead, you do RAG: you take a user query, search your documents for relevant passages, and stuff those passages into the prompt. The foundation model then generates an answer grounded in that extra context. Underneath, you use an embedding model (often itself a foundation model) to convert text into vectors, and you do a similarity search. The developer writes code that splits documents into chunks, embeds them, stores the vectors, and then, at query time, finds the most similar chunks and constructs a prompt like "Answer using this context: [passage]... Question: [query]". The foundation model does the rest.

**Agentic Systems (planners and tool callers).** The newest use is AI agents that plan and use tools. A foundation model is instructed (via prompting or fine-tuning) to output structured steps, like calling a calculator or a database. The model isn't trained to use a specific tool; it just generates text that the developer's code intercepts and executes. For example, a foundation model might output `answer = arange(1, 5)`, and the developer's script runs that Python and feeds the result back into the model. The foundation model is the brain that decides what to do; the code is the hands. When it fails, it's usually because the model hallucinated a tool call — it wrote something syntactically correct but nonsense.

**Everyday Applied ML.** Foundation models aren't just text. Vision models like CLIP or DINOv2 are foundation models for images — a developer uses them to generate embeddings for image search, classification, or similarity. Speech models like Whisper transcribe audio. For a recommendation system, you might use a foundation model to embed user and item descriptions, then feed those embeddings into a separate model that predicts clicks. In fraud detection, you might use a foundation model to summarize transaction narratives before a classifier. Everywhere, the pattern is: embed something with a pre-trained model, then adapt the output.

**What breaks when done badly?** The main failure is treating a foundation model as omniscient. It won't know your private data, it can hallucinate, and it has biases from training. Developers must build guardrails: validate outputs, use RAG for real-time info, and never give a foundation model unconstrained control. The economics lesson from your notes holds: you're renting a model's knowledge, so you must manage its limitations.


---

Read this on the web: https://basavapurushottam.com/learn-ai/foundation-models
