# Modality-Specific Pipelines

Separate ingestion and processing paths per modality before any fusion, because video, text and sensor data need different handling.

**Rests on:** Vision Encoders

*Deep Learning · Multimodal AI*


## Why learn this

Understanding how to process different types of data separately is the foundation for building systems that handle text, images, audio, video, and sensor data together — which is what real-world AI products do every day. Before you can combine information from these different sources into a single understanding, each type must first be cleaned and prepared through its own dedicated path, because each has different speeds, sizes, and quirks. Getting this right means your system can handle video arriving at 30 frames per second while text trickles in as a few words per second, and can keep running even if one sensor drops out. Without this concept, you would try to cram everything into one pipeline and end up with a system that either misses crucial information or crashes when one input is delayed. Mastering this unlocks the rest of the multimodal stack: combining the processed signals from different senses, turning images into standardized numerical summaries for analysis, and creating new images by conditioning on text descriptions — each of which depends on having clean, well-processed inputs from each modality first.


## The intuition

Think of a busy restaurant kitchen with three stations: a grill, a salad bar, and a dessert counter. Each station receives raw ingredients, but they are prepared completely differently. The grill needs meat at room temperature, seasoned, and cooked over high heat; the salad ingredients need washing, chopping, and chilling; the dessert station needs precise measuring and temperature control for baking. You would never mix raw steak with salad dressing or bake a cake on the grill. Each station has its own prep process, its own tools, and its own timing — but at the end, all the completed dishes come together on a plate for the customer. That is exactly how a multimodal AI system works. Text comes in as characters, images as pixels, audio as a waveform, video as a sequence of frames — and each needs its own processing pipeline before you can even think about combining them. The key point is that you cannot skip the individual prep. If you tried to feed raw pixels directly to a language model, it would be meaningless. Each pipeline transforms its raw input into a form the rest of the system can use, and only then do you 'plate' the results together.


## An analogy

Consider a newsroom that covers stories across different formats: a reporter writing an article, a photographer taking pictures, and a videographer shooting footage. The assignment arrives, and each person goes off to work on their own. The reporter interviews sources and types up text; the photographer captures images and edits them; the videographer records sound and video and edits the clips. They each have their own workflows, deadlines, and equipment. But the final story — published online — combines all three: the text article, the embedded photos, and the video player. If the photographer's camera breaks, the article can still be published with just text and maybe a stock photo — the overall story degrades gracefully rather than failing entirely. Critically, the reporter does not wait for the photographer to finish before starting the interview, and the videographer does not rewrite the article to make it fit video. Each medium is handled separately. In a newsroom, the humans coordinate and can adapt to unexpected changes (like sharing notes), whereas in an AI system, the pipelines are code so the 'coordination' is designed in advance — the system does not improvise.


## Definition

A modality-specific pipeline is a dedicated ingestion and processing path for each type of input data (text, image, audio, video, sensor), running independently and in parallel, before any combination of different data types occurs.


## How it connects

This concept sits at the very beginning of the Multimodal AI stack. It builds on nothing (you are here first) and it directly enables the later stages: once each modality has been cleaned and converted into a usable representation, you can move on to Cross-Modal Fusion (combining the processed signals from different senses) and then to higher-level tasks like Vision Encoders (turning images into standardized numerical summaries for analysis).


## The mathematics

The mathematics behind modality-specific pipelines is fundamentally about **arithmetic** — counting, timing, and measuring rates. This concept does not require calculus or linear algebra; it is about understanding that each pipeline operates at its own speed and produces data of a certain size and quality. The core idea is a **rate**: how many items (bytes, frames, samples) arrive per second, and how many can be processed per second. If the processing rate is lower than the arrival rate, the pipeline backs up and you lose data. The mathematics here is about ensuring the numbers work out: arrivals per second must be less than or equal to processing capacity per second. It is also about **synchronisation** — aligning the time stamps of different streams so that, for example, the video frame at time 1.5 seconds corresponds to the audio sample at the same moment. This is simple addition and subtraction of offsets — nothing more than arithmetic. But getting it right is what makes the whole system function.

capacity_per_second = 1 / processing_time_per_item

free_capacity = capacity_per_second - arrival_rate_per_second

aligned_timestamp = original_timestamp + offset


## A worked example

Suppose you have a video pipeline that receives 30 frames per second, and your processing step takes 0.02 seconds per frame. On the text side, you receive 5 words per second, and each word takes 0.001 seconds to process. We need to check if each pipeline can keep up, and if the total load is balanced.

Step 1: Video processing capacity: 1 / 0.02 = 50 frames per second   # If each frame takes 0.02 seconds to process, then in one second you can process 50 frames.
Step 2: Video arrival rate: 30 frames per second (given). Since 30 < 50, video pipeline is not overloaded.   # Capacity (50) is greater than arrival (30), so the pipeline can keep up.
Step 3: Text processing capacity: 1 / 0.001 = 1000 words per second   # Each word takes 0.001 seconds, so in one second you can process 1000 words.
Step 4: Text arrival rate: 5 words per second. Since 5 < 1000, text pipeline is not overloaded.   # The text pipeline has huge spare capacity compared to video.
Step 5: Synchronisation example: video frame at 1.5 seconds, text word arrives at 1.5 seconds. If we shift text by +0.1 seconds to align, the aligned time is 1.6 seconds (not matching). We need to adjust the shift to 0.0 seconds to match.   # Alignment is about adding or subtracting fixed offsets to timestamps so that concurrent events match.

Result: Both pipelines are within capacity, and synchronisation requires careful offset adjustment to align the streams. The video pipeline is closer to its limit (60% utilised) than text (0.5% utilised), so any burst in video frames must be handled by buffering or dropping frames.


## From scratch, in Python

```python
# Plain Python implementation of a modality-specific pipeline
# We'll simulate a simple text pipeline and a video pipeline
# No external libraries needed

import time

# A list of text chunks (simulating incoming text data)
text_chunks = ["hello", "world", "this", "is", "a", "test"]

# A list of video frames (simulating incoming video frames, each as a small list)
video_frames = [
    [1, 2, 3],  # frame 1
    [4, 5, 6],  # frame 2
    [7, 8, 9]   # frame 3
]

# Processing function for text: convert to uppercase
# This is a simple example; real processing would be much more complex
def process_text(text):
    processed = text.upper()
    return processed

# Processing function for video: average the pixel values to get a single value
# Again, a trivial example
def process_video_frame(frame):
    total = 0
    for pixel in frame:
        total = total + pixel
    average = total / len(frame)
    return average

# Main pipeline: process text and video independently
processed_texts = []
for chunk in text_chunks:
    processed = process_text(chunk)
    processed_texts.append(processed)

processed_videos = []
for frame in video_frames:
    processed = process_video_frame(frame)
    processed_videos.append(processed)

# Now we could combine them, but for this example we just print separately
print("Processed text:")
print(processed_texts)
print("Processed video:")
print(processed_videos)

# Real-world would have timing and buffering, but this shows the separation
```


## The same thing, with a framework

```python
import numpy as np

# In practice, you would use libraries like OpenCV for video, or HuggingFace for text.
# Here we just show how numpy can represent a video frame as a matrix.

# A video frame as a numpy array (height, width, channels)
frame = np.array([
    [[255, 0, 0], [0, 255, 0]],
    [[0, 0, 255], [255, 255, 255]]
])

# Convert to grayscale by averaging RGB channels
# This is a typical pre-processing step
gray = np.mean(frame, axis=2)
print(gray)
```


## What the code does

The code defines two separate processing paths: one for text (a list of strings) and one for video (a list of lists representing frames). Each has its own `process_*` function that transforms the data independently. The main loop iterates over each modality's data separately, applying the appropriate function, and stores results in separate lists. This separation ensures that the video pipeline does not interfere with the text pipeline — they run in sequence here but in real systems they would run in parallel. The framework version shows how a video frame might be represented as a NumPy array (a table of numbers) and processed for grayscale conversion. The key point is that each modality gets its own dedicated treatment.


## Common misconceptions

**All data can be treated the same** — Why can't I just feed all my data — text, images, numbers — into one big pipeline and let the model sort it out?

Because each modality has fundamentally different characteristics: text is discrete and sparse, images are dense numeric grids, audio is a continuous waveform. If you mix them, you lose the meaning. Text tokens have no natural order, while pixels have a spatial layout. You cannot apply the same normalisation or feature extraction to all of them. In fact: Each modality must be preprocessed in a way that respects its own structure: text requires tokenization and embedding, images require resizing and normalization, audio requires spectrogram conversion. Only after each is transformed into a common representation can they be combined.

**Streams will naturally stay in sync** — If I just record video and audio at the same time, they will naturally be aligned.

Red herring: different devices have different clocks, and even within one device, processing delays can shift one stream relative to the other. Without explicit timestamp alignment, the video might be 100ms ahead of the audio, which is annoying for humans and catastrophic for machine learning. In fact: Synchronisation is a first-class engineering problem. You must attach timestamps to each item as it arrives, and then apply offset corrections to align the streams. This is done through careful timestamping and, if necessary, cross-correlation methods to find the offset.

**One pipeline failing should stop everything** — If my camera breaks, the whole system should just stop, because I can't do the task without visuals.

In many applications, you can still do useful work with the remaining modalities. For example, a voice assistant can still understand commands without seeing the user's face. Stopping the entire system is overly rigid and wastes the data that is still available. In fact: A well-designed system degrades gracefully: if one pipeline fails, the others continue, and the fusion step works with fewer inputs. This might mean falling back to a text-only response or using a proxy for the missing modality.

**Modality pipelines are only for deep learning** — I only do simple analytics on tabular data, so I don't need to worry about pipelines per modality.

Even with tabular data, you might have numeric columns and categorical columns that need different preprocessing. Numeric columns need scaling, categorical columns need one-hot encoding or label encoding. That is a trivial example of modality-specific pipelines. In fact: The principle applies at all levels of data processing: separate the handling of different types of data so that each can be processed optimally. This is a general engineering practice, not just an AI concept.


## In short

Modalités have different rates, sizes, and quality. Build separate ingestion and processing paths for each — text tokenizes, images resize, audio spectrogram—and only then fuse. Synchronisation is a real engineering problem (timestamps, offsets), and failures should degrade gracefully, not block the whole system. Keep each pipeline independent, but design the fusion step to tolerate missing inputs. Remember: the feature store is the only component that survives a modality change, a model upgrade, or a shift in time horizon — so invest in good data contracts.


## Where this is used

Modality-specific pipelines are everywhere in modern AI, because almost every real system combines multiple types of input.

**Large Language Models (LLMs):** A system like ChatGPT that accepts both text and images must first process each modality separately. Text goes through a tokenizer (converting characters to numbers) and then through the language model's embedding layer. Images go through a vision encoder (like a Vision Transformer) that converts pixels into embeddings. Only after both are converted into the same 'embedding space' can the LLM process them together. The pipeline for images is completely different from text — different code, different preprocessing, different model. If you tried to feed raw pixels to the tokenizer, it would be meaningless.

**Small Language Models (on-device):** On a phone or laptop, you have even stricter constraints. The text pipeline must run quickly with limited memory, so it may use a smaller vocabulary and quantized models. The image pipeline must compress images to low resolution before processing. Each pipeline is optimised independently — text can run on the CPU, while image processing might use the GPU. This separation is what makes it feasible to run multimodal models on devices with limited hardware.

**Retrieval-Augmented Generation (RAG):** In a RAG system, you often have documents that contain both text and images. To retrieve relevant passages, the system must index both. Text is embedded using a text encoder; images are embedded using a vision encoder. The embeddings are stored in a vector database, but the pipelines that create them are separate — you would never embed a sentence with a vision encoder. The retrieval step then searches across both embeddings, which is a fusion step that relies on having clean, separate representations.

**Agentic Systems:** Agents that plan, call tools, and act in loops often receive input from multiple modalities simultaneously — for example, a robot that sees a scene and receives a voice command. The vision pipeline processes the camera feed, and the audio pipeline processes the speech. Each runs in parallel on dedicated hardware (GPU for vision, DSP for audio). The agent then combines the processed outputs from both pipelines to decide on an action. If the vision pipeline fails (camera blocked), the agent must still respond to the audio command — this is graceful degradation, which is only possible because the pipelines are separate.

**Everyday Applied Work:** In a fraud detection system, you might have transaction data (structured numbers) and device fingerprints (strings). The numeric pipeline scales and normalises the data; the text pipeline converts device strings into categorical features. They are processed separately because the numeric processing (e.g., scaling to 0–1) would be meaningless for strings. Later, a fusion step combines them into a feature vector for the model.

**What developers actually do:** They use libraries like OpenCV for video frames, `wave` or `librosa` for audio, `BeautifulSoup` for text extraction. Each pipeline is a separate module—a class or a function that processes its modality. They write code that runs each pipeline in parallel (using threading or async) and then a fusion step that waits for all pipelines to finish. The hardest part is synchronisation: timestamps from different sources must be aligned. They also build in buffering (to handle bursts) and quality checks (to reject corrupt frames). If done badly, you get problems like video and audio out of sync, or the whole system crashing when one input is delayed. Good engineering accounts for these failures gracefully.


---

Read this on the web: https://basavapurushottam.com/learn-ai/modality-specific-pipelines
