# GPT Architecture

A decoder-only transformer trained autoregressively — stacked masked self-attention and feed-forward blocks with normalisation and residuals.

**Rests on:** Feed-Forward Blocks, Positional Encoding

*Deep Learning · Transformers*


## Why learn this

This is the architecture behind ChatGPT, Claude, Gemini, and every other large language model you have used or will build. Understand this system, and you understand how text generation works, why it sometimes makes up facts that sound plausible but are false (hallucinating), what 'context length' means (how much of the conversation or document the model can remember at once), and why a model cannot look ahead in a sentence to cheat. It is also the foundation for smaller models that run on phones, retrieval-augmented generation (using external sources to answer questions), and agentic systems (programs that can take actions). This concept unlocks the rest of the Transformers family: causal attention masking, multi-head attention, positional encoding, and RoPE.


## The intuition

Imagine you are reading a sentence word by word, and at each word you try to predict the next one. You cannot see the future — you only have the words so far and your own memory. GPT works in exactly that way. It is a machine that, given a sequence of words, produces a probability for every possible next word. It does this by passing the words through many identical 'blocks', each of which lets every word look at every earlier word (that is self-attention) and then applies a simple transformation to each word individually (the feed-forward part). The crucial rule is the causal mask: when the model computes the probability for the next word, it is never allowed to peek at words that come after. This is what makes it 'decoder-only' — it only generates, it does not read the whole input at once like a translator. The training is autoregressive: it tries to predict the next word, compares its prediction to the real next word, and adjusts its internal numbers (the weights) to make the prediction better. Over millions of examples, the model becomes excellent at predicting the next word, and that is all it ever does, even when it is answering your question or writing an essay.


## An analogy

Think of a very patient, single-minded storyteller who tells a story one word at a time. They have a rule: they can only use the words already said, never the words they are about to say. At each step, they consider all the words they know (the vocabulary), and give each a score indicating how likely it is to be the next word. They then pick one word (sometimes the most likely, sometimes a random one depending on 'temperature'), and that becomes the next word in the story, and then they repeat. The storyteller's 'brain' is made of many identical offices (the layers). In each office, two things happen: first, each word in the story so far is written on a card, and every card is compared to every earlier card to see how relevant it is — this is self-attention, where words exchange information. Second, each card is transformed by a small 'thought processor' (the feed-forward network) that does not look at other words, only at itself. The office then adds its own contribution to the original cards (residuals) and normalises the numbers (layer normalisation) to keep them stable. The storyteller repeats this office process over and over — say, 96 times for a large model — and at the end, they have a final set of numbers for each word, which they convert into probabilities for the next word. The analogy stops working in one important way: the storyteller is not a person with learned habits; every office is identical in structure, but each has its own set of numbers (weights) that are tweaked during training.


## Definition

GPT (Generative Pre-trained Transformer) is a decoder-only transformer that generates text one token at a time by attending to all previous tokens through stacked self-attention and feed-forward layers, with causal masking, residual connections, and normalisation, trained to predict the next word.


## How it connects

You have not studied any prerequisites yet, so this page starts from absolute basics. But you will soon learn about Causal Attention Masking, which is the single mechanism that makes this a decoder, and Multi-Head Attention, which lets the model attend to different relations. Later you will add Positional Encoding (since attention has no built-in sense of order) and RoPE embeddings. This page gives you the whole picture; the later topics will zoom into the parts.


## The mathematics

The mathematics of GPT rests on two basic operations: multiplying lists of numbers by tables of numbers, and adding those results together. You do not need algebra beyond arithmetic—just multiplication, addition, and division. The key operation is the dot product.

To compute a dot product, take two lists of numbers that have the same length. Multiply the first number in the first list by the first number in the second list. Multiply the second number in the first list by the second number in the second list. Continue this for every position. Then add all those products together. The result is a single number. This number tells you how much the two lists 'agree'. If the result is large, the lists are similar.

In GPT, every word is converted into a long list of numbers called an embedding. When the model wants to know how much one word should pay attention to another, it computes the dot product of their two embeddings. A larger dot product means the words are more related in this context.

The other key operation is matrix multiplication. This is just doing several dot products at once. The model has huge tables of numbers (called weights) that it multiplies with the word vectors to transform them. These weights are adjusted during training so that the final prediction is as accurate as possible.

GPT uses a mechanism called self-attention. For each word, the model creates three lists: Queries (Q), Keys (K), and Values (V). The process works in these steps:

1. Compare every Query to every Key using dot products. This produces a table of scores.
2. Divide each score by the square root of the length of the vectors. This scaling step prevents the numbers from becoming too large, which would make the next step unstable.
3. Convert the scores into probabilities that add up to 1. This is done using a function called softmax. The softmax function takes a list of numbers and turns them into weights where larger numbers get higher weights, but all weights sum to 1.
4. Multiply each Value vector by its corresponding probability weight from step 3. Add these weighted vectors together. The result is the new representation for that word, combining information from other words based on their relevance.

GPT stacks many layers of this process. Each layer also includes residual connections (adding the original input to the output) and normalization (adjusting the scale of the numbers so they stay within a manageable range).


## A worked example

Imagine we have three words: 'I', 'love', 'chai'. Each word is represented by a tiny 2-number vector (just for illustration; real models use hundreds of numbers). Let:
'I' → [1, 0]
'love' → [0, 1]
'chai' → [1, 1]

We are at the step where we compute attention for the word 'chai' to decide which earlier words are relevant. For simplicity, assume the Query, Key, and Value vectors are all the same as the original embeddings in this tiny example.

Step 1: Compute the dot product of 'chai' (Query) with 'I' (Key).
Multiply position by position: 1 * 1 = 1. Then 1 * 0 = 0.
Add them: 1 + 0 = 1.
The score for 'chai' attending to 'I' is 1.

Step 2: Compute the dot product of 'chai' (Query) with 'love' (Key).
Multiply position by position: 1 * 0 = 0. Then 1 * 1 = 1.
Add them: 0 + 1 = 1.
The score for 'chai' attending to 'love' is 1.

Step 3: Scale the scores.
The length of our vectors is 2. The square root of 2 is approximately 1.414.
Divide each score by 1.414:
Score for 'I': 1 / 1.414 ≈ 0.707
Score for 'love': 1 / 1.414 ≈ 0.707

Step 4: Apply softmax to get probabilities.
We need to convert these scores into weights that sum to 1. First, calculate e raised to the power of each score (e is approximately 2.718).
e^0.707 ≈ 2.028
e^0.707 ≈ 2.028
Add these values: 2.028 + 2.028 = 4.056.
Divide each e-score by the total sum:
Weight for 'I': 2.028 / 4.056 = 0.5
Weight for 'love': 2.028 / 4.056 = 0.5
The model considers both words equally relevant.

Step 5: Compute the weighted sum of Values.
Multiply each Value vector by its weight and add them together.
Value for 'I' is [1, 0]. Weight is 0.5. Result: [0.5, 0]
Value for 'love' is [0, 1]. Weight is 0.5. Result: [0, 0.5]
Add the results: [0.5 + 0, 0 + 0.5] = [0.5, 0.5]

Result: The attention output for 'chai' is [0.5, 0.5]. In a real model, this vector would go into the next layer to help predict the word after 'chai'.


## From scratch, in Python

```python
# We will build a miniature GPT-like model in plain Python.
# For this demo, we use a tiny vocabulary of 4 words, and a tiny dimension of 3.

import math
import random

vocab = ["I", "love", "chai", "."]
word_to_idx = {"I":0, "love":1, "chai":2, ".":3}

# Each word gets a random vector (in a real model, these are learned)
random.seed(42)
embedding = {}
for w in vocab:
    embedding[w] = [random.random()-0.5 for _ in range(3)]

# Weights for a tiny attention head (random, not trained)
W_Q = [[random.random()-0.5 for _ in range(3)] for _ in range(3)]  # 3x3
W_K = [[random.random()-0.5 for _ in range(3)] for _ in range(3)]
W_V = [[random.random()-0.5 for _ in range(3)] for _ in range(3)]

# Function to compute the dot product of two lists
def dot(a, b):
    return sum(x*y for x,y in zip(a,b))

# Function to multiply a matrix by a vector (matrix-vector multiplication)
def matmul(M, v):
    return [dot(row, v) for row in M]

# Given a list of word indices (the context), compute attention

def softmax(scores):
    max_s = max(scores)
    exps = [math.exp(s - max_s) for s in scores]
    sum_exps = sum(exps)
    return [e/sum_exps for e in exps]

def attention(context_indices):
    # Compute queries, keys, values for all positions
    queries = [matmul(W_Q, embedding[vocab[i]]) for i in context_indices]
    keys = [matmul(W_K, embedding[vocab[i]]) for i in context_indices]
    values = [matmul(W_V, embedding[vocab[i]]) for i in context_indices]
    
    # For the last position (the one we are predicting), compute attention scores
    last_q = queries[-1]
    scores = []
    for i in range(len(context_indices)):
        # Mask: do not look at future positions (here we only look up to last, so fine)
        scores.append(dot(last_q, keys[i]) / math.sqrt(len(last_q)))
    
    probs = softmax(scores)
    
    # Weighted sum of values
    output = [0.0]*len(values[0])
    for i,p in enumerate(probs):
        for j in range(len(output)):
            output[j] += p * values[i][j]
    return output, probs

# Test with context: "I love chai" (indices 0,1,2)
context = [0,1,2]
out, probs = attention(context)
print("Attention probabilities:", probs)
print("Output vector:", out)

# In a real GPT, this output goes into a feed-forward network, then to a final
# matrix that gives a score for each word in the vocabulary, which is converted
# to a probability via softmax.
# Then the model picks the next word (the one with highest probability), adds it
# to the context, and repeats.
```


## The same thing, with a framework

In PyTorch, you would use the `torch.nn.TransformerDecoderLayer` and stack those layers, or you would use a library like Hugging Face Transformers to load a pretrained GPT model. Under the hood, `nn.MultiheadAttention` does exactly the matrix multiplications and softmax that we wrote by hand. The `causal mask` is passed as an argument. For something like GPT-2, you would do:
```python
from transformers import GPT2LMHeadModel, GPT2Tokenizer
model = GPT2LMHeadModel.from_pretrained('gpt2')
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
inputs = tokenizer("I love", return_tensors="pt")
outputs = model(**inputs)
next_token_logits = outputs.logits[0, -1, :]
next_token_id = next_token_logits.argmax()
```
The library does the entire forward pass: embedding lookup, all the layers of attention and feed-forward, and the final projection to vocabulary scores.


## What the code does

We start by making a tiny vocabulary of four words. Each word has a random vector of three numbers (the embedding). We then create three small matrices (lists of lists) for the queries, keys, and values. The function `dot` computes the dot product of two lists — this is how we compare two words. `matmul` multiplies a matrix by a vector: it takes each row of the matrix and computes the dot product with the vector, producing a new vector. The `attention` function takes a list of word indices (the context). It first computes the query, key, and value vectors for every position by multiplying the embedding of each word by the corresponding matrix. Then, for the last position (the word we are currently predicting), we compute a score against every key (but only up to the last — this is the causal mask; in this code we never look ahead). We scale by the square root of the dimension. The `softmax` function turns these scores into probabilities that sum to one. Finally, we take a weighted sum of the value vectors using these probabilities — that is the attention output. The last part prints the probabilities and the output. In a real model, this output goes through a feed-forward network, then to a final layer that produces a score for each word in the vocabulary, then softmax to get probabilities, and the model picks the next word.


## Common misconceptions

**GPT is a robot that understands language like a human** — GPT is a superintelligent being that truly understands the meaning of words and has consciousness.

GPT processes numbers, not meanings. It has no inner experience. It is a statistical pattern miner. It cannot feel or understand in the human sense. Its impressive output is the result of billions of learned correlations, not understanding. In fact: GPT is a statistical language model. It predicts the next token based on patterns learned from enormous amounts of text. It can produce semantically plausible text without any underlying understanding of the world.

**GPT is trained on everything at once, so it knows the future** — Because GPT was trained on the whole internet, it can use information from the future (tokens after the current one) to make predictions.

The causal mask prevents the model from ever looking forward, both during training and during generation. The training data is presented as sequences, and the model is trained to predict each token using only previous tokens. In fact: GPT is autoregressive: it only conditions on the past. The causal mask hides future tokens. This is why it can generate coherent text one token at a time.

**Each layer does something completely different, like a workflow** — The layers in GPT are like a pipeline where layer 1 does grammar, layer 2 does syntax, etc.

The layers are all identical in structure, and they all perform the same operation (attention + feed-forward). They are not specialised explicitly; they learn to do different functions during training, but they are interchangeable in architecture. There is no pre-defined job for each layer. In fact: All layers are identical blocks, but after training, they often specialise in different types of patterns — some layers attend more to syntax, some to semantics — but this is emergent, not designed.

**Attention is the same as a lookup table** — Attention is like a dictionary lookup where each word is looked up in a fixed table of related words.

Attention is dynamic: the weights are computed based on the content of the query and keys at that moment. The same word in a different context will attend to different words. In fact: Attention computes a weighted mixture of values based on the similarity of queries and keys. It is context-dependent, not a fixed lookup.

**A bigger model always generates better text** — The more parameters a GPT has, the better the output, full stop.

Size helps up to a point, but there are diminishing returns. A larger model requires more data and compute to train, and it can overfit to the training data. Smaller models can be fine-tuned to specific tasks and may be more efficient. In fact: A bigger model does not guarantee better text. Quality depends on training data, objective, and fine-tuning. Larger models can be better, but they also require more resources and can have diminishing returns.

**The context length is the only limit on how much the model remembers** — Once the context is filled, the model forgets everything before that.

While the model only directly attends to tokens within the context window, the weights and the embeddings encode patterns from the entire pre-training corpus. It does not 'remember' the exact text, but it has learned statistical regularities, which is a kind of memory. Also, for long conversations, the model does not recall the exact tokens beyond the context, but it may have distilled information from earlier in the conversation into its internal states, which then influence the last tokens before the window is full. However, in practice, once a token is outside the context, the model cannot attend to it directly. In fact: The context window defines the tokens the model can directly attend to. Information before the context is lost to direct attention, but the model can still be influenced by its learned priors. Managing context is a major engineering challenge.


## In short

GPT is a decoder-only transformer that generates text by predicting the next token one at a time, using causal self-attention to look at all previous tokens, and stacked through many layers with residual connections and normalisation. It is trained autoregressively on massive text corpora. The same architecture powers most modern LLMs, small on-device models, RAG systems, and agentic loops.


## Where this is used

GPT architecture is everywhere in modern AI.

**Large language models** — ChatGPT, Claude, Gemini are all decoder-only transformers. The core of their ability to generate coherent text is the autoregressive loop: the model receives your prompt, converts it to token IDs, passes them through the stacked layers, and outputs a probability distribution over the vocabulary for the next token. It then samples from that distribution (or chooses the most likely token), appends it, and repeats, often until it hits an end-of-text token. When you ask ChatGPT to write a poem, it is not 'thinking' about the whole poem — it is iteratively predicting the next token, using only the tokens it has already generated. The very same architecture that writes essays also writes code, translates languages, summarizes documents, and answers questions.

**Small models on devices** — The same architecture is scaled down to run on your phone or laptop. A model like Llama 3 8B (8 billion parameters) is still a decoder-only transformer, but with fewer layers and smaller dimensions. What changes is the computational cost: every forward pass requires billions of multiplications, and the device must fit them in memory. Developers use quantization (converting numbers from 32-bit to 8-bit or 4-bit) and pruning (removing unused weights) to shrink the model. The causal mask remains essential — the model must generate tokens one at a time, and the mask ensures the GPU can process the whole sequence in parallel during training, but at inference, it is still sequential.

**Retrieval-augmented generation (RAG)** — GPT models are pre-trained on a huge corpus, but they do not have access to a specific company's documents. RAG systems add a separate retrieval step: they search a database of text chunks using vector similarity (embeddings, which are also produced by transformer models), find relevant chunks, and insert them into the prompt before the GPT model generates. The GPT model then generates with that context in its attention window — it can only attend to the tokens in its context, so the retrieved chunks become part of what it has 'seen'. The attention mechanism is what allows it to use the inserted text to answer accurately.

**Agentic systems** — An agent is a loop: the model receives a task, decides which tool to call (e.g., a calculator, a web search), calls it, gets a result, adds it to the context, and generates the next action. The GPT architecture is central because each step of the loop is a new forward pass with the updated context. The causal mask is what prevents the model from seeing the tool output before it exists — it predicts the tool call first, then the environment returns the output, then the model generates the next step. The whole loop is a sequence of autoregressive predictions, each conditioned on the entire history of actions and observations.

**Everyday applied ML** — GPT-style models are used beyond text: they generate images (e.g., DALL-E uses a transformer that generates image tokens), speech (e.g., some text-to-speech systems use transformer decoders), and even tabular data imputation. In fraud detection, a transformer can be trained to predict the next transaction in a sequence, and anomalies are detected by low probability. In recommendation systems, a model can predict the next item a user will click, treating the user's history as a sequence of tokens. In all these, the same pattern applies: encode the input as a sequence, apply causal masking, and predict the next element.

**What developers actually do** — In practice, a developer rarely builds a GPT from scratch; they use a pretrained model from Hugging Face, finetune it on their data, or use an API. They write code like:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")
input_text = "Once upon a time,"
inputs = tokenizer(input_text, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(output[0]))
```
For small models, they might use ONNX to optimise for the device. The biggest failures happen when the causal mask is incorrectly applied — if the model sees future tokens during training, it learns to cheat and the generation collapses. Also, forgetting to reset the model's state between sequences can contaminate the context.


---

Read this on the web: https://basavapurushottam.com/learn-ai/gpt-architecture
