In this article
Context
Our AI agents need memory to maintain context across multi-step tasks and across user sessions. We need to decide how to implement short-term (within session) and long-term (cross-session) memory, balancing recall quality, cost, complexity, and privacy.
Options Evaluated
In-Context Only (no external memory)
Pros
- +Simplest — no additional infrastructure
- +Perfect recall within the context window
- +No data leakage risk between users
Cons
- −Context window limits bound maximum conversation length
- −No persistence across sessions — user must re-establish context every time
- −Token costs grow linearly with conversation length
Vector Database External Memory
Pros
- +Scales to arbitrarily long histories
- +Semantic retrieval — recall relevant memories without storing everything in context
- +Cross-session persistence
Cons
- −Retrieval quality is not perfect — relevant memories may be missed
- −Additional infrastructure (vector DB or pgvector table)
- −Must design read/write policies carefully to avoid privacy leakage
Structured Database + Summarisation
Pros
- +Structured facts stored accurately (user preferences, task history)
- +Summarisation condenses long histories into dense representations
- +Easy to audit what is stored per user
Cons
- −Summarisation loses detail and nuance
- −More complex retrieval logic (SQL + embedding search)
- −LLM summarisation calls add cost and latency
Decision
We implement a two-tier memory architecture: (1) In-context short-term memory with conversation summarisation after 10 turns to manage context growth, and (2) pgvector-based long-term memory for per-user persistent facts (preferences, past topics, correction history). Long-term memory is opt-in and clearly communicated to users. We avoid storing raw conversation transcripts — only extracted key facts and summaries.
Consequences
- •Implement conversation summarisation after every 10 turns using GPT-4o-mini (cost-efficient)
- •Create user_memory table in Neon (pgvector) with: user_id, fact, embedding, category, created_at
- •Retrieve top-5 relevant memories at session start using embedding similarity
- •Memory writes are async (do not block response)
- •Provide users a memory management UI to view and delete their stored facts