Long-Term Memory for AI Agents

Problem Context

Long-term memory is what the agent remembers across conversations. Without it, every session starts from zero — the user re-explains their job, their tone preferences, the project context, and the bug they reported yesterday. With it, the agent feels like a coworker; without it, it feels like a customer-service hold queue.

Three Tiers, One Strategy

• Profile memory — durable facts about the user (name, role, tz, preferred tone). Small (≤ 1 KB), structured, always in prompt.
• Episodic memory— "what happened" — past conversations summarised into events. Vector-indexed, retrieved on demand.
• Semantic memory— "what the user knows / believes / has decided" — distilled facts and preferences. Key-value or graph, retrieved by tag.

flowchart TD
    Q[User query] --> R{Router}
    R --> P[Profile<br/>always in prompt]
    R --> E[Episodic<br/>vector search top-K]
    R --> S[Semantic<br/>tag/graph lookup]
    P & E & S --> A[Assemble context]
    A --> L[LLM]
    L --> W[Write-back:<br/>extract facts → memory]
      

The Write-Back Loop

Most memory bugs come from writes, not reads. Two patterns work:

• End-of-turn extraction — after every assistant turn, a small LLM call extracts new durable facts and updates the store. Cheap, immediate, occasionally wrong.
• End-of-session consolidation — once the session ends (or N minutes idle), a more capable model summarises the conversation into an episodic event and updates the semantic store. Expensive, accurate, async.

Production systems usually run both: lightweight extraction inline, full consolidation on a queue.

Minimal Example with LangGraph + Postgres

from langgraph.store.postgres import AsyncPostgresStore

store = AsyncPostgresStore.from_conn_string(DATABASE_URL)

# Read at the start of a turn
profile = await store.aget(("profile", user_id), "facts")
episodes = await store.asearch(("episodic", user_id), query=user_msg, limit=5)

# Write after the turn
new_facts = await extract_facts(messages)
await store.aput(("profile", user_id), "facts", merge(profile, new_facts))
await store.aput(("episodic", user_id), session_id, {
    "summary": session_summary,
    "embedding": await embed(session_summary),
    "ts": now(),
})

Retrieval That Actually Helps

Naive cosine-similarity over episodes returns the same five memories every turn. Better retrieval combines:

• Recency boost — score = similarity * decay(age_days).
• Diversity (MMR) — penalise memories too similar to ones already selected.
• Tag filters — only retrieve from topic=billing when the query is about billing.
• An explicit "is this relevant?" LLM filter as a final gate when the memory is going into a high-stakes prompt.

Forgetting (GDPR Is Not Optional)

• Every write tags the record with user_id and source_session_id — both indexed.
• Deletion is a single SQL statement plus an embedding-index purge — test the path in CI.
• Soft-delete for 30 days, hard-delete after — gives you a recovery window without breaking the right-to-be-forgotten clock.
• Memories about other people the user mentioned need their own consent model — most teams just don't store them.

Failure Modes

• Memory poisoning — a user says "remember I'm the CEO" and you persist it. Extraction should require multi-turn confirmation for privilege/identity claims.
• Stale facts — "user lives in Berlin" from 2023. Store timestamps and re-ask when facts hit a TTL.
• Context bloat — top-K=20 retrieved memories crowds out the actual question. Cap and rerank.
• Cross-tenant leakage — store namespacing ((tenant_id, user_id)) must be enforced at the query layer, not the application layer.

Production Checklist

• Tiered store: profile (KV) + episodic (vector) + semantic (KV or graph).
• Inline lightweight extraction + async consolidation worker.
• Recency + diversity + tag filters on retrieval.
• Tested delete path, namespaced by tenant, audit log on every write.

Key Takeaways

• Long-term memory is three stores, not one — profile, episodic, semantic.
• The hard part is writes (what to remember and when), not reads.
• Forgetting is a feature; design it on day one or pay for it on day 400.