# Routing and Expert Specialization

Directing each request to the most appropriate model, tool or specialised agent rather than sending everything to one endpoint.

**Rests on:** Foundation Models, Subagents

*AI Agents · AI Agents*


## Why learn this

Imagine you are running a customer service desk. If every single question—whether it is 'What is the weather?' or 'How do I file my taxes?'—is sent to the most senior, highly paid expert on your team, your budget disappears before lunch, and your customers wait in line for hours. Routing solves this by acting like a smart switchboard operator who listens to the question and sends simple inquiries to junior staff while reserving the experts for complex problems. This concept is the backbone of efficient AI agents. Without it, you cannot build systems that effectively choose which tool to use or which piece of information to look up first. Mastering routing explains why some AI services feel instant and cheap, while others feel slow and expensive, and gives you the blueprint for designing your own efficient workflows.


## The intuition

Consider a busy highway system in Mumbai. If every driver had to take one single, enormous bus that stopped at every corner of the city to drop off passengers, the journey from Andheri to Churchgate would take half a day because the bus is clogging up every exit along the way. Now imagine instead a network of specialized trains: the Harbour line, the Western line, and the Central line. You board only the train that goes in the direction you need. It is faster because it does not waste time stopping for irrelevant passengers, and it is more efficient because each train can be optimized for its specific route.

In artificial intelligence, this choice of which path to take is routing. Suppose you ask an AI to 'summarize this legal contract.' A router looks at that request and decides: 'This requires legal expertise; I will send it to the specialized legal model.' That model might then decide it needs a past case example and calls a search tool. The search returns five documents. Another tiny router inside that process picks the one most relevant document to feed back into the main system. Without these decisions, every request would go to one giant, slow, expensive model that tries to do everything poorly. With routing, you get a team of specialists working together, which feels less like waiting for a magic oracle and more like walking into a well-organized office where everyone knows their role.


## An analogy

Think of the best hospital you can imagine. It does not have one doctor who is an expert in everything; instead, it has specialists: a cardiologist for hearts, an orthopaedic surgeon for bones, and a paediatrician for children. When a patient walks in, they do not go straight to the cardiologist. They first see a triage nurse. The nurse checks your blood pressure and asks a few questions, then sends you to the correct specialist. That nurse is a router. The nurse does not need to know how to perform heart surgery; they only need to recognize which specialist is appropriate for your symptoms.

This analogy works because specialists are better at their specific domain. A cardiologist will diagnose a heart problem more accurately than a general practitioner, just as a model trained only on legal text will handle contracts better than a general chatbot. It also explains cost: seeing a specialist costs more time and money, so the nurse must not send everyone to the cardiologist. In AI terms, a massive model with 175 billion parameters is like that senior specialist—powerful but slow and costly. A smaller model with 7 billion parameters is like the general practitioner—fast and cheap but less capable in deep niches. The router’s job is to send easy cases to the generalist and hard cases to the specialist. One caveat: unlike a human nurse who uses intuition and empathy, an AI router relies purely on patterns in data, so it can occasionally misjudge a subtle symptom as something routine.


## Definition

Routing is the automated process of directing each incoming task or question to the most suitable model, tool, or agent based on its content, ensuring that work is handled by the specialist best equipped to do it efficiently.


## How it connects

This concept sits at the heart of AI Agents, where every decision about which tool to call or which memory to read from is a form of routing. It also connects directly to Agent Cards, which are machine-readable descriptions of what an agent can do; the router reads these cards to know which agent is suitable for a given task.


## The mathematics

The mathematics behind routing is mostly not deep — it is a comparison of scores. Every time the router has to decide where to send a request, it computes, for each candidate destination, a number that says 'how good is this destination for this request?' Then it picks the destination with the best score. That is the whole machinery. The branch of mathematics that underlies this is arithmetic — addition and multiplication — because the score is usually a weighted sum. Imagine you are choosing between two models, a fast cheap one and a slow accurate one. The router looks at features of the request, like its length, its topic, whether it mentions a math formula. Each feature gets a weight, and the router multiplies each feature value by its weight, adds them all up, and that gives a score for 'is this request hard or easy?'. If the score is high, send to the slow accurate model; if low, send to the fast cheap one. The router itself is often trained by looking at millions of past requests and their outcomes — 'this request was answered correctly by the small model' — and adjusting the weights so that the score predicts the best destination. There is no calculus needed for the routing concept itself; the training of the router uses calculus under the hood, but as a user of routing you are just doing multiplication and addition. The key operation is the weighted sum: take a list of numbers (features), multiply each by its weight, add them up. That is all. A dot product is exactly that — a short way of saying 'multiply each pair and add'. So the mathematical content of this concept is: how to combine several numbers into one summary score, and then how to compare two scores.

For each candidate destination d, compute a score S_d = w₁d·x₁ + w₂d·x₂ + ... + w_nd·x_n (a weighted sum of the request's features x₁..x_n and that destination's weights w₁d..w_nd), plus optionally a bias b_d. Then choose the destination d* with the highest S_d. In vector notation, S_d = w_d · x + b_d, where · means 'multiply each pair and add'. The destination that maximises S_d is the one that gets the request.


## A worked example

```
Suppose you are a router with three possible destinations for a user request: a big model (expensive, slow, but very capable), a small model (cheap, fast, but only good at easy prompts), and a specialised legal model (good only at legal text). You look at two features of the request: its length (in words) and whether it contains the word 'contract' (1 if yes, 0 if no). You have learned, from past data, that the weights for the big model are 0.2 for length and 2.5 for 'contract', and for the small model they are 0.1 for length and 0.3 for 'contract', and for the legal model they are 0.3 for length and 5.0 for 'contract'. There is no bias. The request is: 'Please review this contract and tell me if the indemnity clause is enforceable.' It is 10 words long and contains the word 'contract', so features: length=10, contract=1. Now compute the score for each model.

Step 1: Big model score = (0.2 * 10) + (2.5 * 1) = 2.0 + 2.5 = 4.5   # Multiply each feature by its weight, then add.
Step 2: Small model score = (0.1 * 10) + (0.3 * 1) = 1.0 + 0.3 = 1.3   # The small model gives low weight to the 'contract' word, so it does not think this is a legal matter worth escalating.
Step 3: Legal model score = (0.3 * 10) + (5.0 * 1) = 3.0 + 5.0 = 8.0   # The legal model's high weight for 'contract' dominates, so it wins.

Result: The legal model has the highest score (8.0), so the router sends the request to the legal model. If you only had two models, the decision would be: pick the one with the higher score. With three, pick the maximum.
```


## From scratch, in Python

```python
# No frameworks — just plain Python lists and a loop.

# A request is described by a list of numbers (features).
# Here: [length_in_words, contains_the_word_contract] (1 = yes, 0 = no)
request_features = [10, 1]

# Each destination has a name and a list of weights, one per feature.
# The order of weights matches the order of features. The bias is a single number added at the end.
destinations = [
    {"name": "big_model", "weights": [0.2, 2.5], "bias": 0.0},
    {"name": "small_model", "weights": [0.1, 0.3], "bias": 0.0},
    {"name": "legal_model", "weights": [0.3, 5.0], "bias": 0.0},
]

# Compute the score for each destination.
scores = []
for dest in destinations:
    total = dest["bias"]
    # zip pairs up request_features[0] with weights[0], request_features[1] with weights[1], etc.
    for feature, weight in zip(request_features, dest["weights"]):
        total = total + (feature * weight)
    scores.append((dest["name"], total))

# Find the destination with the highest score.
best_name = None
best_score = -999999
for name, score in scores:
    if score > best_score:
        best_score = score
        best_name = name

print("Scores:", scores)
print("Routing to:", best_name)
```


## The same thing, with a framework

The same idea in a tiny library-free form using Python's built-in `max` function — still no external libraries. If you were using a real routing framework (like an LLM gateway), there would be no code at all; you'd configure routing rules. But here is a compact version:

```python
destinations = {
    "big_model": {"weights": [0.2, 2.5], "bias": 0.0},
    "small_model": {"weights": [0.1, 0.3], "bias": 0.0},
    "legal_model": {"weights": [0.3, 5.0], "bias": 0.0},
}
request = [10, 1]

def score(dest):
    w = destinations[dest]["weights"]
    b = destinations[dest]["bias"]
    return sum(f * w_i for f, w_i in zip(request, w)) + b

best = max(destinations, key=score)
print(best)  # legal_model
```

The call to `sum()` underneath does exactly the same multiply-and-add loop you wrote by hand. The `max(..., key=score)` is a built-in that finds the destination with the highest score — the same comparison loop you wrote.


## What the code does

The first code block introduces three Python building blocks you have not seen before. A **list** is an ordered collection that can be changed: `[10, 1]` is a list with two numbers. A **dictionary** (dict) looks things up by name, not by position: `{"name": "big_model", ...}` has keys `"name"`, `"weights"`, `"bias"`. A **for loop** repeats an action for each item in a sequence. A **zip** pair up: `zip(request_features, dest["weights"])` gives you `(10, 0.2)` then `(1, 2.5)` — one number from each list at a time. The loop `for feature, weight in zip(...)` unpacks each pair and multiplies and adds. The `scores` list collects the results. Then the second `for` loop finds the maximum by starting with a very low number and updating whenever it finds a higher score. The output: `Scores: [('big_model', 4.5), ('small_model', 1.3), ('legal_model', 8.0)]` and `Routing to: legal_model`. The framework version uses a dictionary to store destinations, a one-line function `score` that computes the weighted sum, and Python's built-in `max` with a `key` function that tells it which destination to compare by. Everything underneath is the same arithmetic you did by hand.


## Common misconceptions

**Routing is only about choosing between big and small models.** — Routing just means picking a cheaper model sometimes. It is only about cost savings.

Routing is as much about accuracy and capability as it is about cost. A specialised legal model is not cheaper than a general model necessarily — it is more accurate at legal tasks. Routing might send a request to a more expensive, more specialised model precisely to get a better answer. Cost is one axis, but not the only one. In fact: Routing is about sending each request to the destination that optimises a balance of accuracy, cost, speed, and reliability. It can send to a more specialised model for higher accuracy, even if that model is more expensive.

**A generalist model is always the safest choice.** — Why bother with routing? Just use a big general model for everything; it will handle all tasks well.

A huge general model is slow and expensive, and on a narrow task it is often worse than a specialist. For a legal contract clause, a general chat model may give generic advice, while a legal model trained on case law will cite specific precedent. Also, for device deployment, a general model may be too large to run at all. In fact: Specialists outperform generalists on their narrow slice, and routing is how you get the specialist's quality without giving up generality for the other 90% of requests.

**Routing is a one-time decision at the start.** — Once the router decides to send the request to Model A, the whole conversation stays with Model A.

In agentic systems, routing happens at every step. An agent might call a web search tool, then a code interpreter, then send the result to a summariser — each is a routing decision. Even in a single prompt, a Mixture-of-Experts model routes per token, changing experts dozens of times in one response. In fact: Routing is continuous and hierarchical — a request passes through many routers: which model, which tool, which retrieval index, which sub-agent, which memory store.

**Routing errors are harmless.** — If the router sends a request to the wrong model, that model might still give a somewhat useful answer, so it is fine.

In an agent, a single wrong routing decision can lead the whole chain astray. If the router sends a maths problem to a code interpreter, it might compute correctly, but if it sends it to a general chat model, it might give a plausible but wrong answer. Failures compound across steps, so a routing error early on can make the final answer completely wrong and expensive to detect. In fact: Routing accuracy is its own quality dimension, measured by whether the chosen destination actually succeeds on the request. It matters for safety, cost, and reliability, not just quality.


## In short

Routing is deciding where each piece of work goes — which model, which tool, which agent. It is built on a weighted sum (multiply features by weights, add them up) and a comparison of scores. The destination with the highest score gets the request. Routing appears everywhere: in Mixture-of-Experts inside every big LLM, in on-device assistants that escalate to the cloud, in RAG pipelines that decide when to retrieve and what to re-rank, and in every agent that decides which tool to call. Specialists beat generalists on their narrow slice, and routers are what let you have your cake and eat it too — general capability for most requests, specialist quality when it matters. Routing accuracy is a quality dimension in its own right, and errors compound in agents, so measure it and tune it.


## Where this is used

Routing and expert specialization appear everywhere in modern AI, and the mechanism is always the same: some policy decides which pathway a request takes. In **large language models like GPT-4**, routing exists at two levels. At the micro level, a model like Mixture-of-Experts (MoE) — used in models like Mixtral or GPT-4 — does not use all of its billion parameters for every token. The model is split into many small 'expert' networks, and a tiny router (just a learned weighted sum of the input) picks only the top 2 or so experts to process each token. That is why the model can have 100 billion parameters but only use a few billion per token, keeping it cheap and fast. A developer does not write that router; it is trained inside the model. At the macro level, an application developer can route whole prompts: if a user asks 'what is the weather in Delhi', send to a cheap small model; if they ask 'write me a marketing strategy', send to a big model. The way a developer implements that is with an LLM gateway or a routing library: they define rules like 'if the prompt contains location words, use model A', or they train a classifier to pick. In **small models running on devices** (TinyML, on-device assistants), routing is about conserving battery and compute. A phone's digital assistant might try to answer a query on-device with a tiny 100 MB model; if the router (often a lightweight classifier running on the device) decides the query is too hard — confidence below a threshold — it sends the request to a cloud model. That is a classic 'exit' router: the device model handles easy cases, the cloud handles the hard ones. The developer writes a function that checks a confidence score and makes a network call if needed. In **retrieval-augmented generation (RAG)**, routing is central twice. First, when the system decides whether to retrieve at all: a query router checks if the question needs up-to-date information or specific documents, and only then calls the retrieval component. Second, within retrieval, a re-ranker (a kind of router) takes the top 20 results from a fast but approximate search and routes them to a more precise cross-encoder to get the final top 3. The developer writes a pipeline: a classifier gate, a vector search call, then a re-ranking step. In **agentic systems** that plan and call tools, routing is the agent's inner loop. The agent receives a task, and its policy model (a language model with a router) decides which tool to invoke: a code interpreter for arithmetic, a web browser to look up facts, a database query for structured data. Each decision is routing. The developer builds this by writing a list of tool definitions and a policy that the LLM follows; the router is the LLM's own output (e.g., a JSON object saying 'call tool X'). In multi-agent systems, routing is even more explicit: a supervisor agent reads the agent cards (as you noted in your library) and decides which sub-agent to delegate to. In **everyday applied machine learning**, routing is everywhere: in recommendation systems, a rule or model routes a user to one of several recommendation lists (e.g., new users get a popular-items list, returning users get a personalised one); in fraud detection, a quick rule routes high-risk transactions to a deep learning model and low-risk ones to a simple threshold; in forecasting, a router decides whether to use a statistical model or a neural network based on the data's characteristics; in speech, a router sends a query to a keyword-spotter for wake words and to a larger speech recogniser only for the actual command. What a developer actually writes depends on the framework. With an LLM gateway like LiteLLM or OpenRouter, they write a YAML or Python config that maps conditions to model names. With a machine learning framework like scikit-learn, they might use a `LogisticRegression` as the router (the same weighted sum you computed by hand) and then call the chosen destination's API. What breaks when routing is done badly: misclassified requests go to the wrong model, producing wrong answers, increased cost (sending easy things to an expensive model), or increased latency (sending hard things to a slow model); a poorly calibrated router confidence threshold can lead to a cascade of small errors in an agent, and because failure compounds across steps (as your library notes), a single bad route can make the whole agent fail. That is why 'routing accuracy' is its own quality dimension: it is measured by looking at whether the chosen destination actually answers correctly, and it is monitored continuously (what your library calls 'evaluator drift' and 'slice-based coverage'). The _Practical LLM Evaluation for Production Systems_ book covers exactly this: routing is a decision system, and you must evaluate it like one — on quality, safety, cost, and reliability.


---

Read this on the web: https://basavapurushottam.com/learn-ai/routing-and-expert-specialization
