# Vision Encoders

Components turning images into embeddings a language model can consume, usually a pretrained vision transformer plus a projection.

**Rests on:** Feed-Forward Blocks, Positional Encoding

**Leads to:** Modality-Specific Pipelines, Object Hallucination, TrOCR and OCR

*Deep Learning · Multimodal AI*


## Why learn this

You want your software to look at an image and understand what is in it. For example, you might need a system that looks at a photo of a document and extracts the text, or an app that identifies a plant from a picture on a smartphone. Vision encoders are the tool that turns those raw pixels into information a computer can actually process. Without them, a program sees only noise; with them, it sees shapes, colors, and objects that can be linked to language.


## The intuition

Imagine you have a photograph of a cat sitting on a rug. A vision encoder’s job is to turn that photograph into a list of numbers that a text-processing system can read. It does this in two clear stages.

First, it uses a special network called a "vision transformer." This network has already been trained to recognize patterns like edges, textures, and shapes from millions of images. When it looks at your cat photo, it outputs a set of features: a mathematical description of what is in the picture, but written in numbers rather than words. 

But those numbers are currently in a format that a language model (the part of the AI that writes or reads text) does not understand. They are like notes written in a foreign code. That is where the second stage comes in: a "projection layer." This is a simple mathematical tool, specifically a linear map (a predictable way to transform numbers while keeping their relationships intact). It takes the features from the vision transformer and projects them into what we call the "embedding space" of the language model. 

Think of it like translating a book from French to English. The vision transformer produces a draft in French, and the projection layer translates that draft into English so the language model can read it. The goal is to ensure that if the image shows a red car, the resulting numbers sit in the same conceptual neighborhood as the words "red car" would for the language model.


## An analogy

Picture a multinational company with two departments: one in Tokyo (vision) and one in New York (language). The Tokyo office writes its reports in Japanese, filled with detailed notes about layout, shading, and edges. The New York office only reads English and needs a summary of the report in a standard format it can act on.

You hire a translator (the projection layer) who takes the Japanese report and turns it into a concise English memo. Each point in the memo is mapped to one of the company’s standard categories. The translator does not change the facts: if the Tokyo report says 'a red car in the parking lot,' the New York memo says exactly that, but structured in English. The translator is trained to do this job well for many different types of reports. Once trained, the translator’s core skills are fixed; you don’t retrain them from scratch every time a new report arrives. Similarly, in AI systems, the vision encoder often learns once to extract visual features, and the projection layer is tuned to align those features with language concepts. In this analogy, the key limitation is that while a human translator understands nuance and context intuitively, the mathematical translation here is strictly a series of weighted sums—calculating how much each number in the input should contribute to each number in the output.


## Definition

A vision encoder is a component that converts an image into a set of numerical representations (embeddings) that a language model can understand. It typically uses a pretrained visual network to identify features in the image, followed by a projection layer that maps those features into the same mathematical space where language models represent words.


## How it connects

This concept sits beside **multimodal AI**, which is the study of systems that process multiple types of data, such as text and images, together. It also connects to **text recognition** (specifically OCR technology), which is the ability of a computer to read and convert printed or handwritten text in an image into editable digital text.


## The mathematics

The key mathematical operation in a vision encoder is the dot product, which is used in the projection layer. The dot product measures how much two lists of numbers align with each other. If you have two lists of the same length, you multiply them entry by entry and then add up the results. For example, if the list A is [2, 1, 3] and list B is [1, 0, 2], the dot product is 2*1 + 1*0 + 3*2 = 2 + 0 + 6 = 8. The result is a single number: 8. This number is large when the two lists have big numbers in the same positions, and small when they don't. In the projection layer, the vision features are a list of numbers (say 768 of them). The projection layer has a matrix—a table of numbers—that it multiplies with the feature list to produce a new list. The matrix is just a set of dot product weights: each row of the matrix is a list that you take a dot product with the input feature list to get one number of the output. So the projection layer is a series of dot products, each giving one number in the language model's embedding space. This is a linear transformation—it only applies multiplications and additions, no squashing—so it changes the coordinates but not the underlying information. The math here comes from linear algebra, which is the branch dealing with lists of numbers and tables of numbers, because a projection is exactly a linear map from one space to another.

Let f be the feature vector from the vision transformer (length d). Let W be the projection matrix (a table of numbers) with rows w<sub>k</sub>. The projected embedding e is computed as: e<sub>k</sub> = w<sub>k</sub> · f = Σ<sub>i=1</sub><sup>d</sup> w<sub>k,i</sub> f<sub>i</sub>. If there is a bias b<sub>k</sub>, add it: e<sub>k</sub> = w<sub>k</sub> · f + b<sub>k</sub>. This is a linear transformation: multiplying a vector by a matrix.


## A worked example

```
Suppose the vision transformer outputs a feature vector of length 3 (to keep it short): [2.0, -1.0, 0.5]. The projection layer is a 2-row matrix (to produce a 2-dimensional embedding for the language model) with weights: row 1 = [0.1, 0.2, -0.3], row 2 = [-0.2, 0.1, 0.4]. There is no bias for simplicity.

Step 1: Row 1 dot product: (2.0 * 0.1) + (-1.0 * 0.2) + (0.5 * -0.3)   # Multiply each feature by the corresponding weight in row 1.
Step 2: = 0.2 + (-0.2) + (-0.15)   # Compute each product.
Step 3: = -0.15   # Sum them: this is the first output number.
Step 4: Row 2 dot product: (2.0 * -0.2) + (-1.0 * 0.1) + (0.5 * 0.4)   # Now multiply by row 2's weights.
Step 5: = -0.4 + (-0.1) + 0.2   # Compute each product.
Step 6: = -0.3   # Sum: this is the second output number.

Result: The projected embedding is [-0.15, -0.3]. This vector is now in the language model's embedding space and can be fed into the model as if it were a token embedding.
```


## From scratch, in Python

```python

import math

# Step 1: The feature vector from the vision transformer (simulated)
feature_vector = [2.0, -1.0, 0.5]

# Step 2: The projection matrix: rows are lists of weights
projection_matrix = [
    [0.1, 0.2, -0.3],  # row 0
    [-0.2, 0.1, 0.4],  # row 1
]

# Step 3: The bias vector (optional, here it's zero but let's include)
bias = [0.0, 0.0]

# Step 4: A function to compute the dot product of two lists
def dot_product(a, b):
    total = 0.0
    for i in range(len(a)):
        total = total + a[i] * b[i]
    return total

# Step 5: Apply the projection: for each row, compute the dot product plus bias
projected = []
for row in projection_matrix:
    value = dot_product(row, feature_vector)
    projected.append(value)

# Add bias (here it's zero, but this is where it goes)
for i in range(len(projected)):
    projected[i] = projected[i] + bias[i]

print(projected)  # Expected: [-0.15, -0.3]

```


## The same thing, with a framework

```python
In PyTorch, a common framework for deep learning, the projection layer is typically implemented as a linear layer: `nn.Linear(in_features, out_features)`. The vision encoder often uses a pretrained vision transformer like `torchvision.models.vit_b_16`, which outputs a feature vector. Then the projection is `proj = nn.Linear(vit_out_dim, language_model_dim)`. Underneath, `nn.Linear` does exactly the dot product we wrote by hand: it stores a weight matrix and a bias, and when called, it computes matrix multiplication. So the plain Python we wrote is what the library call does, just optimized for speed.
```


## What the code does

First, we define the feature vector as a list of numbers—this represents what the vision transformer gave us. Then we have the projection matrix, which is a list of lists: each inner list is a row of weights. The bias is another list of numbers, one for each row, which shifts the output. We define a function `dot_product(a, b)` that takes two lists and returns their dot product: we loop over the indices and accumulate the sum of the products. Then we loop over each row in the projection matrix, compute the dot product with the feature vector, and store the result in a new list `projected`. Finally, we add the bias to each element. The print statement shows the output. This is exactly what the library call `nn.Linear` does, but in slow, visible arithmetic.


## Common misconceptions

**The vision encoder outputs a single picture** — The vision encoder outputs a single number or a compressed version of the image.

It outputs a sequence of embeddings (many numbers), not one. Each embedding corresponds to a patch of the image or a global feature, and together they represent the image's content. In fact: The vision encoder produces a list of embeddings, typically one per image patch, and optionally a final 'class' token that summarizes the whole image.

**The projection layer is just a feature extractor** — The projection layer is just another feature extractor that learns a new representation.

The projection layer's sole purpose is to align the vision feature space with the language model's embedding space. It doesn't learn new features; it translates coordinates. In fact: The projection layer is a linear map that changes the coordinate system of the features to match the language model, so that a dot product between an image embedding and a text embedding is meaningful.

**Higher resolution is always better** — I should always use the highest resolution input to my vision encoder to get the best performance.

Higher resolution improves detail but costs more computation and memory, and may slow down inference so much that the system is unusable in real time. Also, if the encoder was trained at a certain resolution, feeding it higher resolution may not help or may even hurt. In fact: The resolution is a hyperparameter to tune based on your task's need for detail and your deployment constraints. The encoder's resolution bounds what fine detail the model can perceive, so choosing it is a trade-off.

**Frozen means the weights are not trained at all** — When the library says 'freeze the encoder', it means the encoder's weights are kept as a constant and never changed during any training.

Frozen means the weights are not updated during a specific training run, but they were trained previously on a large dataset. They are fixed for the current task to save computation and prevent overfitting, but they are not random. In fact: When you freeze the vision encoder, you keep its weights fixed (like constants) while training only the projection and possibly the language model. The encoder still produces useful features, but it doesn't adapt to the new task.


## In short

A vision encoder takes an image and produces embeddings for a language model. It has two parts: a pretrained vision transformer (usually frozen) and a projection layer that maps the features into the language model's space using a linear transformation (dot products). The resolution limit of the encoder determines what fine details the model can perceive, and you typically only train the projection layer, not the whole encoder, to save resources.


## Where this is used

Vision encoders are everywhere in multimodal AI. Here's the mechanism and what a developer actually writes for each scenario.

- **Large language models (LLMs) with vision**: Models like GPT-4V or LLaVA accept images as input. A vision encoder (like a CLIP ViT) converts the image into a sequence of embeddings. Those embeddings are then passed to the LLM through a projection layer that maps them into the same space as the text token embeddings. The LLM then reasons over the combined sequence. A developer doesn't write the vision encoder from scratch; they use a library like HuggingFace Transformers: `from transformers import AutoProcessor, AutoModelForVisionText2Text` and then feed the image through the processor to get pixel values, then through the model. The projection layer is a minor part of the architecture, but it determines how well the visual information is understood.

- **Small models running on devices**: On a phone, you can't run a massive vision transformer. So you use a smaller vision encoder, like MobileNet or EfficientNet, which are designed to be fast and efficient. The projection layer then has to be small too. The trade-off is accuracy versus speed: a smaller encoder can't capture as much fine detail, so the model might mistake a dog for a cat in low light. A developer would use TensorFlow Lite or PyTorch Mobile and convert the model to a compact format. The key is that the projection layer is kept simple (just a matrix multiplication) so it doesn't add much latency.

- **Retrieval-augmented generation (RAG)**: In RAG, you might have a database of images, and you want to find the one most similar to a user's query. You use a vision encoder to embed all the images, and a text encoder to embed the query. Then you compute similarity (often dot product) between the query embedding and image embeddings to find the best match. This is how systems like Google's image search or visual search in shopping apps work. A developer would use a pretrained CLIP model, which has both a vision and text encoder that are trained to align. They'd compute embeddings for all images and store them in a vector database, then use a similarity search to find the nearest neighbor.

- **Agentic systems that plan and call tools**: An AI agent that can look at a screenshot and decide to click a button needs to understand the image. The vision encoder gives it a description of the screen as embeddings, which the agent's language model uses to reason about what to do. For example, in UI automation, the agent might be told to 'find the login button'. The vision encoder produces embeddings that let the model identify the button. A developer might use a tool like Playwright combined with a vision-language model, but the core is that the vision encoder turns the screen into a representation the model can act on.

- **Everyday applied machine learning**: In medical imaging, a vision encoder can analyze X-rays to detect anomalies. In manufacturing, it can inspect products for defects. In agriculture, it can estimate crop yield from drone images. In each case, the developer uses a pretrained vision encoder (like ResNet or ViT) and then trains a small classification head on top. The projection layer here is that final classification layer, not the alignment layer we discussed, but the principle is the same: take features, then map them to a decision. The developer writes something like `model = torchvision.models.resnet18(pretrained=True)` and then replaces the final layer with a new one for their number of classes.

A developer reaches for a library like HuggingFace Transformers, PyTorch, or TensorFlow. They write code that loads a pretrained model, processes the image into a tensor, and extracts embeddings. What breaks when done badly: if the projection layer is not trained properly (e.g., you don't align the dimensions), the language model might ignore the image or produce gibberish. If the vision encoder is too small or low-resolution, fine details are lost, causing errors in tasks like OCR or object detection. If you freeze the vision encoder but the image domain is very different from what it was trained on (e.g., medical x-rays vs natural photos), the features may be useless, and you need to fine-tune the encoder. The choice of resolution is critical: a 224x224 image loses small text, but a 1024x1024 image is expensive. So the developer must balance.

In summary, a vision encoder is not just a feature extractor; it is the crucial link that lets language models 'see', and its design choices affect everything from accuracy to speed.


---

Read this on the web: https://basavapurushottam.com/learn-ai/vision-encoders
