# Model Serving APIs

The HTTP or RPC surface exposing a model, including schema validation, health checks and versioned endpoints.

**Leads to:** Model Registry and Versioning

**Rests on nothing.** A place to start.

*MLOps and Platform · MLOps*


## Why learn this

A trained model on your computer is like a chef who knows exactly how to cook but has never seen a customer. Without an interface, you cannot give it new data to process, nor can you see if it is still working correctly. When you connect the model to an application—whether that is a website showing recommendations or a system detecting fraud—you need a bridge. This bridge takes the user's information, delivers it to the model, and returns the result. Building this bridge allows you to update the model without stopping the service, check that the system is healthy, and ensure the predictions remain accurate over time.


## The intuition

Imagine a busy restaurant. The kitchen (the model) has a recipe for predicting loan approval based on three numbers: income, debt, and years at the current job. Customers (applications) do not walk into the kitchen; they stand at a counter (the API). 

When a customer hands a slip of paper to the counter with the words 'high income', the counter stops them. It says, 'I need three numbers, not words.' This ensures that only correct information enters the kitchen. This check is called schema validation.

The counter also checks if the kitchen is open. If the chefs are on a break or the stove is broken, the counter tells the customer, 'We cannot take orders right now.' This immediate feedback prevents customers from waiting for an answer that will never come. This status update is what we call a health check.

Furthermore, the counter needs to know two distinct things about the kitchen's state. Is the kitchen *ready* to take new orders? (Are the chefs at their stations and the ingredients prepared?). And is the kitchen *alive*? (Is the power on?). Readiness means 'I can accept work.' Liveness means 'I am not frozen or dead.'

Suppose the restaurant updates its recipe. Instead of stopping service, they might slowly let 5% of customers get the new prediction while everyone else gets the old one. This is a canary release: testing a change on a small group before showing it to everyone. If the new recipe causes errors, the counter detects this drift from expected results and alerts the manager. 

Finally, because the kitchen has changed its recipe slightly over time, the counter must remember what format of information it used during training versus what format users send now. Mismatched formats are known as training-serving skew, and the API is where we spot and fix these mismatches before bad predictions go out.


## An analogy

Think of the system as a secure bank vault. The model is the safe inside. The API is the heavy steel door with a keypad and a sensor array. You do not touch the money or the algorithms directly; you interact only through the door. When you press a code, the door checks if the numbers make sense (schema validation). If the power goes out, the door signals it is locked down (liveness). If the door opens to let you in, it signals that the interior is accessible and safe for transactions (readiness). The vault does not improvise; if you enter letters instead of numbers, the door refuses entry instantly. Unlike a human guard who might get tired or confused, this steel door follows strict physical rules every single time without fail.


## Definition

A model serving API is the standardized interface that allows external applications to send data to a trained model and receive predictions back, while simultaneously managing access control, input validity checks, and system health monitoring.


## How it connects

This concept sits next to Model Registry and Versioning, which is the library where you store different copies of your model so the API can switch between them when needed. It also relates to Monitoring and Alerting, which is the practice of watching the data flowing through this API to spot when predictions start looking unusual or when system errors increase.


## The mathematics

This concept does not rest on a new branch of mathematics. It uses only arithmetic and the idea of a function, which you already know from everyday life: a function takes an input and produces an output. Here, the input is a set of numbers (the features) and the output is the model's prediction (a number or a category). The API does not change the mathematics of the model — it just wraps it in a layer that handles communication. But there is one mathematical idea that is useful: the idea of a 'request' as a tuple of values. A tuple is just an ordered list — like (income, debt, years_at_job). The API checks that the tuple has the right length and the right types, and then the model processes it. No new maths is needed; it is all about structure and validation.

The only formula is the model's prediction function, which is not particular to APIs: y_pred = f(x1, x2, x3) where f is the trained model. The API does not alter this — it only ensures that (x1, x2, x3) are valid and that y_pred is returned in a standard format.


## A worked example

```
Imagine a simple loan-default model that takes three inputs: income (in lakhs), debt (in lakhs), and years at current job. The model outputs a probability of default. The API expects a request in the form: {income: 5, debt: 2, years_at_job: 4}. We'll trace what happens when this request comes in.

Step 1: Request arrives: {income: 5, debt: 2, years_at_job: 4}   # This is the JSON body sent by the client.
Step 2: API checks schema: income>0, debt>=0, years_at_job>=0 and all are numbers   # If any check fails, the API returns a 400 error and does not call the model.
Step 3: Model computes: raw_score = 0.5*5 - 1.2*2 + 0.3*4 = 2.5 - 2.4 + 1.2 = 1.3   # This is the weighted sum using the model's weights.
Step 4: Probability = 1 / (1 + exp(-1.3)) ≈ 0.786   # The math is just arithmetic for this example; the API does not change it.
Step 5: API returns: {score: 0.786, decision: "default"}   # The response is a structured JSON that the client can parse.

Result: The API successfully served the request, validated the input, ran the model, and returned a structured response. The entire process is just arithmetic and data packaging.
```


## From scratch, in Python

```python
import json
import datetime

# We'll build a tiny API in plain Python, without any web framework.
# This is to see exactly what happens under the hood.

# The trained model is just a function that takes three numbers and returns a prediction.
def loan_probability(income, debt, years_at_job):
    # These are the model's learned weights (example values)
    w1 = 0.5
    w2 = -1.2
    w3 = 0.3
    bias = -0.5
    raw_score = w1 * income + w2 * debt + w3 * years_at_job + bias
    # sigmoid function to get a probability between 0 and 1
    probability = 1 / (1 + (2.71828 ** (-raw_score)))
    return probability

# A simple function that simulates an HTTP request handler.
# This is what a real API would do: receive a request, validate it, call the model, return a response.
def handle_request(request_string):
    # Step 1: Parse the input (like the HTTP request body)
    try:
        data = json.loads(request_string)
    except json.JSONDecodeError:
        return {"status": 400, "error": "Bad JSON"}

    # Step 2: Schema validation - check that required fields are present and have correct types
    required_fields = ["income", "debt", "years_at_job"]
    for field in required_fields:
        if field not in data:
            return {"status": 400, "error": f"Missing field: {field}"}
    if not isinstance(data["income"], (int, float)) or data["income"] <= 0:
        return {"status": 400, "error": "Income must be a positive number"}
    if not isinstance(data["debt"], (int, float)) or data["debt"] < 0:
        return {"status": 400, "error": "Debt must be a non-negative number"}
    if not isinstance(data["years_at_job"], (int, float)) or data["years_at_job"] < 0:
        return {"status": 400, "error": "Years at job must be a non-negative number"}

    # Step 3: Call the model
    prob = loan_probability(data["income"], data["debt"], data["years_at_job"])

    # Step 4: Build the response
    response = {
        "status": 200,
        "score": round(prob, 4),
        "decision": "default" if prob > 0.5 else "no_default",
        "served_by": "model_v1",
        "timestamp": str(datetime.datetime.now())
    }
    return response

# Simulate a request
request_body = '{"income": 5, "debt": 2, "years_at_job": 4}'
print(handle_request(request_body))
# Output: {'status': 200, 'score': 0.786, 'decision': 'default', 'served_by': 'model_v1', 'timestamp': '...'}

# Test with a bad request
bad_request = '{"income": "high"}'
print(handle_request(bad_request))
# Output: {'status': 400, 'error': 'Missing field: debt'}  (and so on)
```


## What the code does

This code is a simplified API, but it covers the key parts: parsing input, validating schema, calling the model, and returning a structured response. In real life, you would use a web framework like FastAPI or Flask, which handle HTTP routing, JSON parsing, and error codes for you. Underneath that framework, it does exactly what this code does: parse the request, validate, call the model, return a response. The main difference is that a framework lets you define routes like `/predict` and `/health`, and it deals with concurrency and network details. So when you use `fastapi.APIRouter`, understand that it is just a helper for the same logic you see here.


## Common misconceptions

**The API is just a container – it doesn't need validation** — The model is the smart part, so I'll just put a simple endpoint that calls it. Validation is optional.

Without validation, malformed input reaches the model, which can cause errors, infinite loops, or unsafe predictions. It also wastes compute. The API is a boundary where the first line of defence lives. In fact: Validation is a core responsibility of the API. It checks that the input has the right shape and type before the model runs.

**Health checks are optional** — I can skip health endpoints because my model is always up.

Models can be down due to resource limits, crashes, or deployment issues. Without health checks, orchestration systems (like Kubernetes) can't automatically restart or route around problems. In fact: Health endpoints (liveness and readiness) are standard practice. They tell the orchestrator whether to send traffic to this instance.

**Versioning is just a nicety – I can always deploy the new model** — I'll just replace the model when I retrain. Nobody needs old versions.

Without versioning, you can't roll back if a new model performs worse. You also break any client that expects a different response format. In fact: Versioned endpoints let you keep multiple versions live and switch traffic slowly (canary). It's key for release safety.

**Schema enforcement is for beginners – real systems don't need it** — My model can handle any input, so I don't need to check types.

Models assume a specific input format. A string instead of a number can cause silent errors or exceptions, and it's hard to debug. In fact: Schema enforcement catches malformed input at the boundary, protecting the model and giving clear error messages to users.

**The API is the model – they're the same thing** — When I talk about the API, I mean the model itself.

The API is a separate layer that manages communication. The model does the prediction. Understanding the difference is essential for debugging. In fact: The API is the interface; the model is the computation. They are separate components that can be deployed and scaled independently.

**If the API returns 200, the prediction is correct** — A successful response means the model was right.

A 200 just means the request was processed. The model could be wrong, or the input could be mis-scaled, or the model could be badly calibrated. In fact: A 200 indicates that the API successfully handled the request. Model quality is a separate question, monitored through metrics and drift detection.


## In short

The model serving API is the gatekeeper between the world and your model. It validates inputs, checks health, and routes to the right version. Without it, your model is inaccessible and unmanageable. Remember: validation belongs in the API, health checks keep you alive, and versioning lets you roll back safely.


## Where this is used

Where is a model serving API used? Everywhere a model needs to be accessible.

**Large language models (LLMs):** LLMs like GPT-4 are exposed through APIs. When you chat with a model on a website, the website sends a request to an API endpoint like `/v1/completions`. The request includes a prompt (the text you typed) and parameters like temperature. The API validates the prompt (e.g., it's a string, not too long), checks that the model is loaded and ready, and then streams back the generated text. The API also handles versioning: if OpenAI releases a new model version, they put it behind the same endpoint but with a different version number, like `/v1/chat/completions` for the chat model vs `/v1/completions` for the base model. The API is what allows millions of users to call the model without each having to download it.

**Small models on devices:** When you run a model on your phone (e.g., for keyword spotting like 'Hey Siri'), there isn't a remote API — the API is local, inside the app. But it still serves the same purpose: it accepts input (audio data), validates it, runs the model, and returns a result. For tiny models that run on microcontrollers, the API is even simpler, but the principles hold. The difference is that the API is not over HTTP; it might be a function call in C. But conceptually, it's the same.

**Retrieval-augmented generation (RAG):** In RAG, you often have a model that retrieves relevant documents and then generates an answer. The serving API might have an endpoint like `/rag/query` that accepts a user question. The API validates the question, calls the retrieval system (which is itself a model — an embedding model), gets the relevant documents, and then passes them to the generation model. The API is the glue that orchestrates multiple model calls. If the retrieval model is down, the API should return a 503 error. Versioning matters a lot here: if you update the embedding model, the API might need to route to a different version of the retrieval index, and the endpoint might be versioned like `/v1/rag` vs `/v2/rag`. The API is where you can also log traces, as in MLflow Tracing, to debug what happened.

**Agentic systems:** An agent that plans and calls tools is exposed via an API, but the API is often the agent's interface to the outside world. For example, a customer support agent might have an endpoint `/agent/respond` that accepts a user message and returns a response. Behind the API, the agent might call a tool (a weather API, a database query), and the API handles the orchestration. The API is the surface for health checks: if the agent is stuck in a loop, the readiness endpoint might report unhealthy. Also, the API is used to rate-limit requests and to provide a consistent interface for the front-end.

**Everyday applied ML (recommendation, vision, fraud, speech):** In a recommendation system, the API takes a user ID and returns a list of recommended items. The API validates the user ID, fetches features, calls the model, and returns the list. In fraud detection, the API receives a transaction and returns a fraud probability. This is a high-throughput API, and schema validation is critical — a malformed transaction can cause the model to misbehave or crash. In computer vision, the API might accept an image (as base64) and return object detections. The API must validate that the image is in the right format, resize it if needed, and call the model. In speech recognition, the API accepts audio bytes and returns text. In all these cases, the API is the only way the model is accessed, and it is where schema validation, health checks, and versioning are implemented.

**What a developer actually writes:** A developer uses a web framework like FastAPI to define endpoints. For example, they write `@app.post('/predict')` and define a function that takes a JSON body, validates it (using Pydantic models), calls the model function, and returns a response. They also define a `/health` endpoint that returns a simple JSON like `{'status': 'ok'}` when the model is loaded and ready. For versioning, they might include a version in the URL (`/v1/predict`) or in the request body. They rely on libraries like `pydantic` for schema validation, `uvicorn` to run the server, and `mlflow` to load the model. If the schema validation is done badly (e.g., not checking types), a user can send a string where a number is expected, and the model will crash or produce garbage. If health checks are absent, the orchestrator might route traffic to a dead model, causing downtime. If versioning is not handled, you cannot roll back to a previous model without breaking clients.


---

Read this on the web: https://basavapurushottam.com/learn-ai/model-serving-apis
