Agent Memory Architecture: Why Your AI Forgets After 3 Sessions
Most AI agents lose context after a few sessions. The context window is your entire "memory" — and when the session ends, it's gone. Agents repeat mistakes, user preferences reset, and multi-step workflows break across sessions.
We run 15 production agents across content, operations, and lead generation. By session 4, they'd lost every decision, preference, and fix from the first three. So we built a persistent memory layer that maintains context across 50+ sessions with sub-second retrieval.
Here's the architecture that made it work.
The Problem: Stateless Agents
Most agent frameworks treat memory as an afterthought. The standard fix — stuffing more context into the prompt — hits token limits fast and degrades response quality as context grows.
What you actually need is an external memory store that the agent queries on demand. Think of it like a human reaching for notes instead of trying to memorize everything.
The Three-Layer Architecture
Layer 1: Session Summaries
After each session, compress the full conversation into structured facts:
def extract_session_summary(transcript: str) -> dict:
return {
"decisions": [...], # What was decided
"preferences": [...], # What the user likes/dislikes
"facts_learned": [...], # New information discovered
"open_questions": [...] # Unresolved items
}This compresses a 30K-token session into 200-500 tokens of high-signal memory. Four fields is all you need to start.
Layer 2: Semantic Indexing
Embed each summary using a fast embedding model. We use `mxbai-embed-large` (1024-dim, MTEB 64.68). Store embeddings in a vector database alongside BM25 text indices for hybrid retrieval.
At session start, the agent queries: "What do I know about this user and their current project?" The top-k results get injected into the system prompt.
Why hybrid? Pure vector search misses exact keyword matches. Pure BM25 misses semantic similarity. We weight 70% vector, 30% BM25 for the best of both.
Layer 3: Temporal Decay
Not all memories are equal. Recent interactions matter more than month-old ones. We use an FSRS-inspired importance scoring function:
def compute_importance(memory, days_old):
base = memory.access_count * 0.3 + memory.edge_count * 0.7
decay = 0.95 ** days_old
return base * decayMemories that are frequently accessed or highly connected to other memories resist decay. Isolated, old memories naturally fade — just like human memory.
Results After 6 Weeks
| Metric | Before | After | Change |
|--------|--------|-------|--------|
| Task completion rate | 60% | 94% | +34 pp |
| Context re-establishment | 4.2 min | 120ms | -99.9% |
| User satisfaction | 6.1/10 | 8.8/10 | +2.7 |
| Memory store size | — | 1,900 chunks | — |
| Retrieval latency | — | 120ms (CPU) | — |
Five Common Mistakes
- Storing everything — Your memory store isn't a database dump. Only extract decisions, preferences, and facts. Raw conversation logs are noise.
- No decay function — Without temporal weighting, your agent treats a 6-month-old preference the same as yesterday's explicit correction.
- Single retrieval strategy — Pure vector search misses exact keyword matches. Pure BM25 misses semantic similarity. Use hybrid retrieval.
- Skipping compression — Storing full session transcripts burns your embedding budget and pollutes search results. Compress first, embed second.
- No edge tracking — Isolated memories should decay. But memories connected to many other facts (high edge count) should persist longer. Track the graph.
Getting Started
- Pick an embedding model that runs locally (`mxbai-embed-large` or `bge-m3` for 1024-dim)
- Build the session summary extractor with 4 fields: decisions, preferences, facts, open questions
- Set up hybrid retrieval with both vector and BM25 indices
- Add the decay function and run a weekly consolidation pass
- Measure before/after on task completion rate — that's your north star metric
The Takeaway
The best memory system is one that forgets the right things.
Most teams over-engineer the storage and under-engineer the retrieval. Start with what your agent actually needs to recall at session start, and work backwards from there.
