# Reasoning Models

Models trained to produce extended intermediate reasoning before answering, trading latency and tokens for accuracy on hard problems.

**Rests on:** Foundation Models, Tokenization

*LLMs and Generative AI · Large Language Models*


## Why learn this

This concept is the gateway to understanding how modern AI solves hard problems — problems that need multiple steps of thinking rather than a single guess. Once you master this, you'll be able to follow how models like ChatGPT can solve math word problems, write multi-step code, or plan a trip, and you'll know why sometimes it's better to ask a cheap model quickly than to pay for a thinking model. It also sets you up to understand the next topics in your library: prompt engineering (how to ask the model to think step by step), retrieval-augmented generation (when to look up facts instead of reasoning), and AI agents that plan and call tools in loops. Without this, those topics will feel like magic. With it, they'll feel like engineering.


## The intuition

Think of two kinds of people you ask for advice. The first gives you an answer instantly, based on a gut feeling. If you ask them a simple question — 'what's 2+2?' — they're great. But if you ask them a complex one — 'should I buy a house now or wait five years?' — they might give you a confident but poorly thought-out answer. The second person doesn't answer right away. They say 'let me think', then they work through the problem step by step: income, savings, interest rates, inflation, job stability, and so on. Only after they've reasoned through all the steps do they give you an answer. The second person is like a reasoning model. It has been trained to spend extra time (and extra computational effort, called 'compute') to produce a chain of intermediate thoughts before arriving at the final answer. This extra 'thinking' dramatically improves performance on problems that require multiple logical steps, like solving a math puzzle, writing a piece of code that has to coordinate several functions, or planning a route with many constraints. The trade-off is that it costs more — more time and more money (because you pay for the tokens it uses to write its thoughts). So the key skill in using these models is knowing when to use them. For simple questions, a quick model is cheaper and just as good. For hard ones, a reasoning model is worth the cost. This concept teaches you how that trade-off works and how to make that decision.


## An analogy

Here's an analogy carried as far as it goes: imagine a mathematics teacher who has two ways of marking an exam. In the first way, she reads the final answer and gives a mark based only on whether it's right or wrong. In the second way, she requires every student to show their working — every step of the calculation — and she gives partial credit for correct intermediate steps, even if the final answer is wrong. The second system is more expensive (it takes her longer to mark), but it does two things: first, it encourages students to write down their steps, which forces them to think more carefully; second, it gives the teacher much more information about where the student went wrong. Now, imagine a model trained the same way. Instead of just predicting the final answer, it's trained to produce a chain of reasoning tokens first, and that chain is rewarded during training if it leads to the correct final answer. The model learns that writing out 'let's think step by step' actually improves its chance of getting the answer right. Where the analogy breaks down: a human teacher can check the logic of each step, but an AI reasoning model has no such external check — it's self-taught, so its 'reasoning' can sometimes be a chain of confabulations that look plausible but lead to a wrong answer. Also, the extra tokens are not just a scaffold — they are the very substance of the model's output, and they carry a real monetary cost. So unlike a teacher, the model doesn't care about showing its work for the sake of the reader; it only cares about producing a chain that statistically improves its final prediction.


## Definition

A reasoning model is an AI system trained to produce a series of intermediate reasoning tokens (a 'chain of thought') before giving its final answer, trading extra computational cost and latency for higher accuracy on problems that require multiple logical steps.


## How it connects

This concept sits inside your library's parent concept, Large Language Models (LLMs). LLMs are next-token predictors — they just guess the most likely next word. Reasoning models are a special kind of LLM, one that has been trained to use its next-token predictions to generate a 'thinking' process before the final answer. The key idea from your notes on LLMs that matters here is the 'context window'. The model's entire working memory for a call is the context window — everything it sees, including the chain of thought it generates, must fit inside that window. If the chain gets too long, it runs out of room and may forget the original question. Also, your notes on temperature and sampling are relevant: reasoning models are often run at low temperature to make the chain of thought more deterministic, but the sampling process is still stochastic, so even the same prompt can give a different chain each time. Finally, this connects to the quality-cost-latency tradeoff from your notes: reasoning models are deliberately trading cost and latency for quality on hard tasks. The neighbouring notes you have on 'LLM Nondeterminism' and 'Context Window' are both constraining factors you'll need to manage when using reasoning models.


## The mathematics

The mathematics behind reasoning models is not a fancy new operation — it's built on plain arithmetic and probability. The core idea is this: when a language model generates text, it's making a series of choices, one token at a time. For each choice, it assigns a probability to every possible next token based on what it has already produced. The reasoning model simply uses this same mechanism to generate a longer sequence of tokens — the chain of thought — before switching to the final answer. The mathematics of how it learns to do this is the mathematics of probability: the model is trained to maximize the probability of the correct final answer given the input question. The chain of thought is just an intermediate sequence that helps it do that. So, when you read that a model 'assigns a probability to each token', that's not algebra — it's just counts. If you have a training text of 1,000 words, and the word 'the' appears 50 times, then the probability of picking 'the' as the next word is 50 divided by 1000, or 0.05. That's ratio arithmetic. Reasoning models are trained on millions of such examples, and they learn to produce chains that make the final answer more likely. The 'math' you need to understand is: more steps often means higher accuracy, but each step adds a token, and tokens cost money and memory — that's where the trade-off becomes a decision, not a formula.

Let C be the cost per token. Let N-simple be the number of tokens needed for a simple question (often 1). Let N-hard be the number of tokens needed for a hard question with reasoning (often 10 or more). Let A-no-reason be the accuracy without reasoning. Let A-reason be the accuracy with reasoning.

The cost of an answer is C multiplied by the number of tokens. The value of reasoning is the increase in accuracy, which we call Delta-A. Delta-A equals A-reason minus A-no-reason.

The decision rule is: use reasoning when the cost of N-hard tokens is less than the benefit of Delta-A in your context. In plain arithmetic: if N-hard times C is less than the value of Delta-A, then reason.


## A worked example

Let's make it concrete with a tiny example. Suppose a model can produce a chain of thought. For a simple question, it might need only 1 token of thinking before giving the answer. For a hard question, it might need 10 tokens. Each token costs ₹0.001 (a fraction of a rupee). The model's accuracy on a hard problem is 40% without reasoning, and 80% with a 10-token chain.

Step 1: Cost without reasoning: 1 token × ₹0.001 = ₹0.001   # We pay for every token the model generates, including the thinking tokens.
Step 2: Cost with reasoning: 10 tokens × ₹0.001 = ₹0.010   # The chain of thought adds 9 tokens, so the total cost is 10 times higher.
Step 3: Accuracy gain: 80% - 40% = 40 percentage points   # The reasoning model is twice as accurate on this hard problem.
Step 4: For a simple question, accuracy is 95% both ways, chain of 1 token.   # If the question is trivial, the chain of thought adds no accuracy gain.

Result: On a hard problem, reasoning costs 10 times as much but doubles accuracy. On a simple problem, reasoning costs 10 times as much and adds nothing. The decision is: spend the extra money only when the problem is hard enough that the accuracy gain is worth it.


## From scratch, in Python

```python
import random

# This is a toy model that simulates reasoning.
# It does not use real neural networks - it just shows the structure.

def simple_model(question_difficulty):
    # A very simple model: guesses the answer based on a random chance.
    # For easy questions, it gets it right 95% of the time.
    # For hard questions, it only gets it right 40% of the time.
    if question_difficulty == 'easy':
        return random.random() < 0.95
    else:
        return random.random() < 0.40

def reasoning_model(question_difficulty):
    # A reasoning model: first 'thinks' by generating a chain of 10 tokens,
    # then gives the answer. The thinking improves the chance on hard questions.
    # We simulate the cost by counting tokens.
    tokens_used = 1  # initial token for the answer
    if question_difficulty == 'hard':
        # chain of thought: we pretend it needs 10 extra tokens
        # that would be written to the context, but for simplicity we just count
        # we also improve the accuracy because the chain helps
        tokens_used += 10
        # worse accuracy than easy, but better than simple model on hard
        correct = random.random() < 0.80
    else:
        # easy question: no extra thinking needed
        correct = random.random() < 0.95
    return correct, tokens_used

# Simulate a set of questions
questions = ['easy', 'hard', 'easy', 'hard', 'hard']

print("Running simple model:")
for q in questions:
    correct = simple_model(q)
    print(f"  Question: {q}, correct? {correct}")

print("\nRunning reasoning model:")
for q in questions:
    correct, tokens = reasoning_model(q)
    print(f"  Question: {q}, correct? {correct}, tokens used: {tokens}")

# Calculate total cost assuming each token costs 0.001 rupees
total_cost = 0
for q in questions:
    # using reasoning model
    if q == 'hard':
        total_cost += 11 * 0.001  # 1 answer token + 10 reasoning tokens
    else:
        total_cost += 1 * 0.001
print(f"\nTotal cost for reasoning model on all questions: Rs. {total_cost:.3f}")
```


## What the code does

The code above is a simulation, not a real neural network, but it captures the essence. First, we define a simple_model that takes a question_difficulty ('easy' or 'hard') and returns True if it gets the answer right. It uses random.random() to generate a number between 0 and 1, and if that number is less than the accuracy threshold (0.95 for easy, 0.40 for hard), it returns True. Second, we define reasoning_model. For easy questions, it just gives the answer (1 token) and has the same high accuracy. For hard questions, it 'thinks' by generating 10 extra tokens (we just count them) and then gives an answer with improved accuracy (0.80). The function returns both the correctness and the number of tokens used. Finally, we simulate a list of questions or 'easy' or 'hard', run both models, and compute the total cost for the reasoning model. This shows the trade-off: the reasoning model gets more right, but it costs more.


## Common misconceptions

**Reasoning models always give correct answers** — If the model thinks step by step, it should never make a mistake.

The chain of thought is just a sequence of tokens; the model can make an error in an intermediate step and that error propagates to the final answer. Also, the model only maximises the probability of the final answer, not the logical validity of the steps. In fact: Reasoning models are more accurate than non-reasoning models on hard problems, but they are not infallible. You should still verify the output for critical tasks.

**Reasoning is free** — Why doesn't every model reason? It's just writing extra text, so it should be cheap.

Every token a model generates has a cost, both in compute and in money. The chain of thought adds many tokens, so the total bill can be 10 times higher for a single query. In fact: Reasoning is a deliberate trade-off: you pay more for higher accuracy on hard problems. Use it sparingly.

**Reasoning models think like humans** — The chain of thought is the model's actual conscious reasoning, just like a human thinking aloud.

The model has no consciousness. It has learned statistical patterns that produce chains that often lead to correct answers, but those chains can be nonsensical to a human while still being useful, or vice versa. In fact: The chain of thought is a useful inner mechanism, but it is not human-like reasoning. It is a sequence of tokens that the model has learned to produce because they improve its success rate.

**Reasoning helps for every question** — I should always ask for a chain of thought, just to be safe.

For simple retrieval questions ('What is the capital of France?'), the model already knows the answer with high confidence. Adding a chain of thought adds time and cost with no accuracy gain. In fact: Reserve reasoning for multi-step problems — math, logic, planning — and use a quick model for simple lookups.

**The chain of thought is the final answer** — The chain of thought is what I'll show the user.

The chain is an internal working; it can be messy or contain steps that would confuse a user. It is not written to be presented, and many APIs hide it by default. In fact: You should treat the chain as a tool for the model, not as a user-facing explanation. Generate a separate summary if you need to show reasoning.

**All reasoning models are the same** — If I use any model with 'reasoning' in its name, I get the same quality and cost.

Different models trade off reasoning depth, speed, and cost differently. Some are designed for high-effort reasoning, others for moderate effort. The quality varies by domain. In fact: Test different models and settings on your specific task to find the right balance of cost, latency, and accuracy.


## In short

A reasoning model is an LLM trained to generate a chain of intermediate tokens before its final answer, trading extra cost and latency for higher accuracy on multi-step problems. The key skill is routing: use it only when the problem is hard enough that the accuracy gain justifies the added tokens. It is central to agentic systems that plan and call tools, and a supporting player in RAG and simple QA. The mathematics is just probability and arithmetic — no new formulas. Remember the trade-off: every token counts, and a long chain can cost ten times a direct answer.


## Where this is used

Reasoning models are used in practice wherever a task benefits from careful, multi-step thinking rather than a single quick guess. In large language models (like the ones powering ChatGPT or Claude), reasoning is often triggered automatically when the model detects a hard question — or you can explicitly ask for it by prompting with something like 'let's think step by step'. The model generates a chain of thought inside its context window, and only after that chain does it produce the final answer. Developers building on top of these models will often set a flag or use a specific model endpoint that enables reasoning. For example, in the OpenAI API, you might call a model like 'o1' or 'o3' which is specially trained for reasoning, and the API returns not just the final answer but also the reasoning steps (or hides them, depending on the configuration). When you ask such a model to solve a math word problem, it will write out equations and intermediate calculations in its chain before giving the final number. The cost is that every token in that chain counts toward your bill, so a long chain on a hard problem can be 10-20 times more expensive than a direct answer.

For small models running on a phone or laptop (like TinyML models or small language models that run locally), reasoning is usually too expensive. These small models have limited memory and compute, so they cannot generate long chains of thought. Instead, developers use smart routing: they send easy questions to the small on-device model, and only when the model is uncertain or the task is clearly hard do they send it to a cloud-based reasoning model. This is exactly the quality-cost-latency tradeoff from your notes. For example, a voice assistant on your phone might use a tiny model for 'what's the weather today?' (so it answers instantly and for free), but if you ask 'plan a 3-day itinerary in Goa with budget constraints', it will route that to a reasoning model in the cloud, pay the extra cost, and wait a few seconds.

In retrieval-augmented generation (RAG), reasoning models are useful for the 'generation' part but also for the retrieval itself. RAG combines a retriever (which finds relevant documents) with a generator (which produces the answer). A reasoning model can be used as the generator, and its chain of thought can help it decide which parts of the retrieved documents are actually relevant. But there's a cleverer use: some systems use a reasoning model to generate the search query itself — for example, instead of asking 'what are the side effects of aspirin?', the model reasons: 'the user is asking about drug interactions, so I should search for 'aspirin and warfarin interaction' rather than just 'aspirin side effects'. This is called 'query expansion' or 'reasoning-based query generation'. In practice, a developer would call the reasoning model first to transform the user's question into a better search query, and then the retrieval is done with a simpler embedding model.

In agentic systems — the ones that plan, call tools, and act in loops — reasoning models are the engine. An agent is given a goal (e.g., 'book a flight, hotel, and restaurant for a conference in Mumbai'). It must break that into subtasks, call a flight API, then a hotel API, then a restaurant API, and verify each step. A reasoning model is perfect for this because its chain of thought lets it plan the sequence in advance and adapt if a call fails. Instead of just naively calling APIs in order, it reasons: 'hotel availability depends on the flight dates, so I need the flight booking first'. This is called 'planning' and it's a direct use of the chain of thought mechanism. In code, an agent loop might look like: (1) call the reasoning model with the user's goal, (2) get back a plan as a series of tool calls, (3) execute each tool call, (4) feed the results back into the model, (5) repeat until the goal is achieved. The reasoning model's extra tokens are spent on thinking through the plan, which is exactly what makes the agent reliable.

In everyday applied machine learning — like recommendation systems, fraud detection, or forecasting — reasoning models are less common because these tasks are often about pattern matching from historical data, not about multi-step logic. A model that recommends movies doesn't need to 'think' for 10 steps; it just needs to find similar items. However, there are edge cases. In fraud detection, a reasoning model could be used to generate a natural-language explanation for why a transaction is flagged — the chain of thought can be used to produce a human-readable justification. Similarly, in forecasting, a reasoning model might be used to reason about a time series with known external factors (e.g., 'unemployment rate increased, so sales may drop, but it's a holiday season, so maybe not'). But in practice, most of these systems use simpler, faster models because the latency and cost of reasoning are not justified.

What does building software with this look like? A developer uses an API. For example, with OpenAI's API, you might use the 'chat.completions' endpoint with a model like 'o1' and set 'reasoning_effort' to 'high' or 'low'. The response will include a 'reasoning' field that contains the chain of thought (or it might be withheld for safety). The cost is based on the total tokens generated, including the reasoning tokens. If you don't want to see the chain, you can hide it, but it still counts toward your bill. When done badly — e.g., using a reasoning model for every request, even simple 'what time is it?' — you waste money and increase latency. When done well, you route based on difficulty, perhaps using a simple heuristic (like the length of the user's question) or a small classifier model that predicts whether the question needs reasoning. This routing is itself a product decision, exactly as your notes on the quality-cost-latency tradeoff say.


---

Read this on the web: https://basavapurushottam.com/learn-ai/reasoning-models
