Case studies, and what they cost
Real architecture calls made building this platform — the problem, every option weighed, what got picked, and what it committed us to. Written so you can borrow the reasoning, not just the answer.
13 case studies recorded
Multi-Tenant Data Isolation Strategy for RAG Pipelines
The problem
As we expand from a single-tenant learning platform to supporting teams and organisations, we need to isolate content data between tenants in our RAG pipeline. A data leak where…
3 options weighed
Content Database Migration — Legacy PostgreSQL to Neon
The problem
Our platform content (topics, domains, glossary terms, patterns, decisions) had been stored in a legacy PostgreSQL setup. As our content grows and we add AI features requiring…
3 options weighed
Testing Strategy for LLM-Powered Features
The problem
Traditional unit and integration tests are insufficient for AI features because LLM outputs are non-deterministic. We need a testing strategy that gives us confidence in AI…
4 options weighed
Tool Integration Protocol — MCP vs Custom Tool Calling
The problem
Our AI agents need to call external tools — content search, user data lookup, web scraping, code execution. We must decide whether to build custom tool integrations using OpenAI…
2 options weighed
Agent Memory Architecture — In-Context vs External Memory
The problem
Our AI agents need memory to maintain context across multi-step tasks and across user sessions. We need to decide how to implement short-term (within session) and long-term…
3 options weighed
AI Observability Platform — LangSmith vs Phoenix vs Custom OpenTelemetry
The problem
We need an observability platform to trace LLM calls, evaluate output quality, debug RAG retrieval, and monitor production AI quality over time. Without observability, we are…
3 options weighed
LLM Streaming Architecture — SSE vs WebSocket vs Polling
The problem
Our AI chat and content generation features need to stream LLM responses to users rather than waiting for full completion. We need to choose between Server-Sent Events (SSE),…
3 options weighed
Agentic Workflow Framework — LangGraph vs Direct Implementation
The problem
We are building agentic features — multi-step AI workflows that plan, use tools, and iterate. We need to decide whether to use LangGraph (stateful graph-based orchestration),…
3 options weighed
Embedding Model Selection
The problem
We need an embedding model for our RAG pipeline. The model determines retrieval quality, vector dimensions (affects storage), and API cost.
4 options weighed
Orchestration Framework — LangChain vs Semantic Kernel
The problem
We need an orchestration framework for building RAG pipelines, agent workflows, and tool-calling chains. The two leading options are LangChain (Python/JS) and Semantic Kernel…
3 options weighed
Primary LLM Selection — GPT-4.1 vs Claude 3.5
The problem
We need to select a primary LLM for our platform's AI features (article assistance, search, code generation). The choice affects cost, quality, and vendor dependency.
3 options weighed
Vector Database Selection
The problem
We need a vector database for our RAG pipeline. Options range from managed services (Pinecone, Weaviate Cloud) to integrated solutions (pgvector in our existing PostgreSQL).
3 options weighed
RAG vs Fine-Tuning for Domain Knowledge
The problem
We need to give our LLM domain-specific knowledge about AI engineering tools, patterns, and best practices. The model must provide accurate, up-to-date answers grounded in our…
3 options weighed