# Agent Memory Systems

The stores an agent reads and writes across turns — working context, episodic history, semantic facts and procedural skills.

**Rests on:** Agent Loop, Context Window

**Leads to:** Conversation State

*AI Agents · AI Agents*


## Why learn this

You are going to build AI agents — systems that don't just answer one question but carry on a conversation, call tools, and act over several steps. Without memory, every turn starts from zero: the agent forgets what you said two messages ago, loses the fact that it already looked up the train schedule, and cannot learn from its own mistakes. This concept gives you the vocabulary and the architecture to design agents that remember what matters and forget what doesn't. It unlocks: Agentic Workflows (where memory is what lets a multi-step plan survive), Multi-Agent Systems (where agents share what they know), and AI Security (where memory is what an attacker can poison or steal).


## The intuition

Imagine you are a waiter at a busy restaurant. A table orders food, then asks for water, then asks for the bill — you need to remember the whole exchange to serve them correctly. Your memory is not one single thing. You hold in your working memory what they just asked for and what you are in the middle of doing. You have episodic memory of this particular table: they were here last week and complained about the slow service, so you check on them. You have semantic memory of facts that are always true: the restaurant only accepts card payments, the chef is vegetarian. And you have procedural memory of how to do things: how to carry a tray, how to write a bill. A computer agent that talks to people and acts in the world needs exactly the same four kinds of memory, but stored in a way a program can read and write. When you ask an AI to 'remember that I prefer window seats', that fact has to go somewhere. When an agent is halfway through booking a flight, the partially-filled form has to live somewhere. When the agent calls a weather API, the result has to be kept so it can use it in the next turn. All of that is agent memory, and designing it well is the difference between an agent that feels thoughtful and one that feels like it has amnesia.


## An analogy

Think of a chef in a busy kitchen. The chef has a printed recipe card for the dish they are cooking right now — that's working memory: what the current task needs, held close and discarded when the plate is plated. The chef also has a notebook of special requests from regular customers ('Mr. Gupta likes his biryani extra spicy') — that's semantic memory: facts about the world that persist across visits. The chef remembers that yesterday a new oven was installed and the cooking times are now shorter — that's episodic memory: events from the past that can inform what to do today. And the chef knows, without thinking, how to make a roux or julienne a carrot — that's procedural memory: skills that were once learned and are now automatic. For a computer agent, the analogies map directly: working memory is the context window of a single conversation turn, semantic memory is a database of facts the agent has been told or has learned, episodic memory is a log of past interactions and tool results, and procedural memory is a set of skills or code routines the agent can call. The analogy breaks down in a key way: a human chef's memory is stored in a brain that is always attached and always refreshed. A computer agent's memory is often stored in separate files or databases, and the agent has to explicitly decide what to write down and what to read back. If the designer forgets to save something, it's gone forever. If they save everything, the agent drowns in irrelevant details. The art of agent memory is the retention policy: what to keep, what to forget, and when to bring something back.


## Definition

Agent memory is the set of stores an agent reads from and writes to across turns, typically divided into working context (what is needed right now), episodic history (what happened in the past), semantic facts (general knowledge), and procedural skills (how to do things).


## How it connects

This concept sits directly under AI Agents. Your notes on the Agent Loop describe the decision–act–observe cycle; memory is what provides the 'context' for the next iteration. It also depends on Tool Use, because the results of tool calls are exactly the kind of episodic memory that needs storing. It leads naturally to Agentic Workflows (where memory is what makes a long-running task coherent) and to Multi-Agent Systems (where agents share memory via communication protocols). You have also noted in your library that memory is a central component of agent systems, not an add-on — and that the retention policy (what to keep and what to forget) is the key design decision.


## The mathematics

Agent memory systems do not require advanced mathematics, but they do rest on one fundamental idea: a lookup. When an agent needs to remember something, it does not scan every byte it has ever stored — that would take too long. Instead, it stores items in a structured way and retrieves them by a key or by a measure of similarity. The mathematics here is simple counting and comparison. Counting: how many items are in memory, and how many can be retrieved in a reasonable time. Comparison: when you store a new fact, you might want to know if it is the same as an old fact, or how close it is. The only arithmetic involved is comparing numbers: if each memory item has a timestamp, you can sort them and take the most recent; if each has a relevance score, you can take the highest. There is no calculus and no algebra beyond the idea that numbers can be ranked. The branch of mathematics here is arithmetic and basic logic — ordering, counting, and comparison.

For retrieving the most recent item from a set of items with timestamps: 

```
most_recent = argmax_i (timestamp_i)
```

For retrieving the most relevant item with a relevance score above a threshold θ:

```
candidates = {i : relevance_i > θ}
most_relevant = argmax_i (relevance_i) for i in candidates
```

Here, `argmax_i` means 'the index i that gives the largest value of the expression inside the parentheses'. `timestamp_i` and `relevance_i` are the timestamp and relevance score of the i-th memory item. `θ` (the Greek letter theta) is a threshold value you choose — for example, 0.5 in the example above.


## A worked example

```
A simple agent memory store has 5 entries, each with a timestamp (in minutes since the session started) and a relevance score (on a 0 to 1 scale). The entries are: (A: timestamp 2, relevance 0.9), (B: timestamp 10, relevance 0.4), (C: timestamp 15, relevance 0.7), (D: timestamp 20, relevance 0.1), (E: timestamp 25, relevance 0.8). The agent needs to retrieve the most recent entry and the most relevant entry.

Step 1: Most recent: compare timestamps. 25 > 20 > 15 > 10 > 2, so E is most recent.   # We rank the timestamps from highest to lowest to find the newest item.
Step 2: Most relevant: compare relevance scores. 0.9 > 0.8 > 0.7 > 0.4 > 0.1, so A is most relevant.   # We rank the relevance scores from highest to lowest to find the most important item.
Step 3: If we want the most recent entry with relevance above 0.5, we filter: A (0.9), B (0.4), C (0.7), D (0.1), E (0.8) → relevance > 0.5 gives A, C, E. Then among those, sort by timestamp: 2, 15, 25 → most recent is E.   # Here we apply two filters: first keep only entries with relevance above 0.5, then pick the one with the highest timestamp.

Result: The most recent entry is E (timestamp 25). The most relevant entry is A (relevance 0.9). The most recent entry with relevance above 0.5 is E (timestamp 25, relevance 0.8).
```


## From scratch, in Python

```python
python
# A simple in-memory episodic memory store
# We'll store memories as a list of tuples: (timestamp, content)

# First, let's understand the Python building blocks this code uses:
# - A list is an ordered collection that can be changed. Example: [1, 2, 3]
# - A tuple is an ordered collection that cannot be changed after creation. Example: (1, 'hello')
# - A dictionary maps keys to values. Example: {'name': 'Alice', 'age': 30}
# - A function is a reusable block of code. Example: def add(a, b): return a + b

# We'll use a list named 'memory' to hold all our episodic memories
memory = []  # starts empty

def remember(content, timestamp):
    """Store a new memory (content) with the given timestamp."""
    # Append a tuple of (timestamp, content) to the memory list
    memory.append((timestamp, content))

def recall_most_recent():
    """Return the memory with the highest timestamp (the most recent)."""
    if not memory:
        return None
    # Start with the first memory as the most recent
    most_recent = memory[0]
    # Loop over all memories to find the one with the highest timestamp
    for item in memory:
        if item[0] > most_recent[0]:
            most_recent = item
    return most_recent[1]  # return the content (ignore the timestamp)

def recall_filtered_by_relevance(relevance_scores, threshold):
    """Return memories with relevance above the threshold, sorted by relevance.
    relevance_scores is a dictionary mapping memory content (or index) to a score."""
    # Build a list of (score, content) for items that pass the threshold
    candidates = []
    for content, score in relevance_scores.items():
        if score > threshold:
            candidates.append((score, content))
    # Sort the candidates in reverse order (highest score first)
    candidates.sort(reverse=True)
    # Return only the contents (scores removed)
    return [content for score, content in candidates]

# ----- Example usage -----
# Add some memories
remember('The customer asked for a window seat.', 1)
remember('The flight to Delhi leaves at 8pm.', 2)
remember('The customer prefers vegetarian food.', 3)

# Print the most recent memory
print(recall_most_recent())  # prints: The customer prefers vegetarian food.

# Give relevance scores (as a dictionary)
relevance = {
    'The customer asked for a window seat.': 0.9,
    'The flight to Delhi leaves at 8pm.': 0.4,
    'The customer prefers vegetarian food.': 0.7
}
# Retrieve memories with relevance above 0.5, sorted by relevance
print(recall_filtered_by_relevance(relevance, 0.5))
# prints: ['The customer asked for a window seat.', 'The customer prefers vegetarian food.']

```


## The same thing, with a framework

```python
In a real agent framework, you might use a vector database like FAISS or Chroma. Instead of storing plain text, you store 'embeddings' — lists of numbers that represent the meaning of the text. The lookup happens by mathematical similarity rather than by exact match. The library call `collection.query(query_embeddings=[vec], n_results=3)` does under the hood exactly what our `recall_filtered_by_relevance` does, but with thousands of items and using vector arithmetic (like the dot product) instead of simple scores. The principle is the same: you have a store, you have a way to score items by relevance, and you return the top few.
```


## What the code does

The code defines two functions. `remember` takes a piece of text (the content) and a number (the timestamp) and stores them together as a single tuple inside a list called `memory`. Think of the list as a shelf where each slot holds one memory card that has the timestamp written on the front. `recall_most_recent` looks through that entire shelf, compares the timestamps, and returns the content of the card with the highest timestamp — the newest one. The second function, `recall_filtered_by_relevance`, does something a bit more sophisticated. It takes a dictionary where each key is the content (a string) and each value is a relevance score (a number). It first filters the dictionary to keep only entries whose score is above the threshold, creating a list of `(score, content)` tuples. Then it sorts that list in reverse order — highest score first — and finally throws away the scores, returning only the contents in that order. The `reverse=True` argument means the sort puts the biggest score first, so the most relevant memory comes out first. This is exactly how a real memory system decides what to show the agent: it ranks candidates by relevance and takes the top few.


## Common misconceptions

**Memory is just storing the conversation history** — I can just paste the whole chat log into every prompt and that's memory, right?

That is working memory only, and it has hard size limits. Real agent memory involves selectively storing, summarising, and retrieving relevant information — not just cramming everything into the context window until it overflows. In fact: Memory is an architecture with multiple stores (working, episodic, semantic, procedural), each with its own read/write patterns and retention policies. Effective memory is what you choose to keep, not everything you could keep.

**All memories are equally important** — Once I store a fact, I should always use it.

Old facts can become obsolete (a store closes, a rule changes), and irrelevant memories can confuse the agent or consume context. A memory system needs to rank by recency and relevance, and sometimes deliberately forget. In fact: Memories are ranked and filtered. You keep the most recent and the most relevant, and you discard or summarise the rest.

**Memory is an optional add-on** — I can build a great agent without any memory — it works fine on single questions.

On a single question, memory is irrelevant, but the moment an agent takes multiple steps, calls tools, or has a conversation, it needs to remember what it did and what it learned. Without memory, it repeats actions and loses context. In fact: Memory is a central component of agent systems, not an add-on. It is what makes multi-turn interactions and long-horizon tasks possible.

**Memory is always in the model's weights** — The LLM already remembers everything it was trained on, so I don't need external memory.

The model's weights encode general knowledge from training, but they do not remember this specific user, this conversation, or this session's tool results. In fact: Model weights are like a person's general knowledge; the agent needs external memory for the specifics of each interaction.

**Finetuning is the same as memory** — To make an agent remember a new fact, I should finetune the model on that fact.

Finetuning changes the model's weights permanently and expensively; it is meant for broad knowledge, not for individual user-specific facts. It also risks catastrophic forgetting. In fact: For per-session or per-user facts, use external memory (prompt injection, vector DBs, databases). Finetuning is for changing the model's general skill set, not for remembering one user's preference.


## In short

Agent memory is not a single storage bin. It has four distinct kinds: working context (what is in front of you right now), episodic history (what happened in this session), semantic facts (general truths that persist), and procedural skills (how to do things). Good memory design is about what to keep, what to forget, and how to retrieve — by recency and relevance. Without memory, agents are goldfish; with poor memory systems, they drown. Master these four stores and the retrieval policies around them, and you can build agents that carry context across turns, learn from their own actions, and adapt to their users.


## Where this is used

Agent memory is the backbone of almost every modern AI system, and it appears in different forms at every scale.

**Large language models (LLMs).** By themselves, an LLM has no memory beyond its context window. Everything it 'knows' about a conversation has to be in the prompt you send it. When you chat with ChatGPT, the entire conversation history is re-sent on every turn — that's the model's working memory, but it is ephemeral and bounded. The moment the conversation exceeds the context window, older parts are silently dropped. This is why LLM applications spend so much effort on summarising past turns or selectively retaining the most important pieces. For example, a support chatbot might keep the full text of the last 10 messages, but if the conversation goes longer, it summarises the earlier ones into a few bullet points to fit the window.

**Small language models on devices.** On a phone or a laptop, the context window is even smaller because of memory and compute limits. A keyboard app's autocomplete model has no external memory at all — it uses the current sentence and maybe the last few words. But a personal assistant running locally might keep a small file of facts about the user (the semantic memory) to personalise suggestions, while relying on the on-device model's ability to use that compact memory. The developer writes code to load that file into the prompt before every inference, and to update it when the user states a new fact.

**Retrieval-augmented generation (RAG).** This is where semantic memory becomes explicit and large. RAG systems embed a big collection of documents (or chat logs) into vectors and store them in a vector database. When the user asks a question, the agent converts the question into a vector, finds the most similar stored vectors, and pulls those texts into the prompt. That is a retrieval from semantic memory. The developer's work is deciding what goes into the store, how to chunk long documents, and what the retrieval threshold should be — too low and the prompt fills with irrelevant text, too high and the agent can't answer. For marketing or support, this is often the single most important decision in the system.

**Agentic systems that plan and call tools.** This is where all four memory types come together. Working memory holds the current plan and the result of the last tool call. Episodic memory holds the history of what has been tried and what happened. Semantic memory holds facts learned from earlier conversations. Procedural memory holds the code for each tool the agent knows how to call. When an agent books a flight, it first uses episodic memory to remember that it already checked the departure city, then uses semantic memory to recall a rule like 'the customer prefers morning flights', and holds the provisional booking in working memory until the user confirms. The developer implements this by writing the agent loop to explicitly read from and write to these stores: after each tool call, the result is written to episodic memory; before each decision, relevant episodic memories and semantic facts are injected into the prompt.

**Everyday applied machine learning.** In recommendation systems, a user's past behaviour is episodic memory; their profile is semantic memory; and the model that converts features into scores is procedural memory. In fraud detection, each transaction is checked against both the user's typical pattern (semantic) and recent anomalies (episodic). In forecasting, past observations are stored and re-weighted when new data arrives. The developer's job is to decide the retention window: how far back to keep data, and how to summarise or discard it.

**What a developer actually writes.** In Python, a developer might use ChromaDB or FAISS to store embeddings, and a framework like LangChain or LlamaIndex to wire up a simple memory loop. For a chat agent, they might write:

```python
from langchain.memory import ConversationBufferMemory
memory = ConversationBufferMemory()
memory.chat_memory.add_user_message("I prefer window seats")
memory.chat_memory.add_ai_message("Noted!")
```

Underneath, this stores the messages in a list and injects them into the prompt on every call. For a more advanced agent, they might use a vector store and a similarity search. What breaks when this is done badly: if the retention policy keeps too much, the prompt grows huge and costs more or exceeds the context window; if it keeps too little, the agent forgets important constraints. If the memory is stored without proper access control, an attacker can read it (privacy leak) or inject false memories (poisoning). The developer is essentially designing the agent's amnesia on purpose.


---

Read this on the web: https://basavapurushottam.com/learn-ai/agent-memory-systems
