What we actually use in production

The default architecture for grounding LLM responses in private or current knowledge. Mature tooling, well-understood failure modes.

Cache LLM responses by semantic similarity of prompts. GPTCache, Redis with vector extensions, and Momento all support this. Proven cost reduction at scale.

Use JSON schema / grammar-constrained decoding (Instructor, Outlines, guided JSON) to get reliable structured output. Eliminates fragile regex parsing.

Prompting models to reason step-by-step before answering. Robust across all frontier models for reasoning-heavy tasks.

LLMs calling structured external functions. Native support in GPT-4o, Claude 3, Gemini 2.5. Foundation for agent systems.

Techniques like LLMLingua reduce context by 3–20× with minimal quality loss. Significant cost savings for long-context workloads.

Sample multiple reasoning paths and majority-vote the answer. High accuracy improvement; high token cost — evaluate the tradeoff.

Parameter-efficient fine-tuning via low-rank adapters. Use when RAG + prompting can't close the quality gap. Requires labelled data and eval infrastructure.

Multi-step agents that plan and execute tool chains. Powerful but failure modes compound — build robust observability before shipping.

Use a smaller draft model to speculatively generate tokens, verified by the larger model. 2–3× throughput improvement. Operational complexity is real.

The reference platform for most production AI workloads. Strong ecosystem, reliable uptime, predictable pricing.

Claude 3.5 Sonnet leads on long-context and coding tasks. Excellent structured output support. Strong alternative to GPT-4o for production.

Gemini 2.5 Pro leads on long-context benchmarks. Flash variants offer best cost/performance for high-throughput workloads.

Streaming-first AI primitives for Next.js / React. Unified interface across providers. Excellent DX for UI-layer AI features.

Focused RAG framework with good data connectors and indexing primitives. Less magic than LangChain, better for RAG-specific workloads.

Strong open-weight model hosting. Mistral Small 3.1 punches above its weight. Viable alternative when vendor lock-in is a concern.

High abstraction, large ecosystem, but significant magic. We prefer lower-level SDKs for production. Use for rapid prototyping only.

Multi-model managed service with IAM integration. Good for AWS-native orgs; abstraction layer adds latency and reduces control.

Managed vector database. Serverless tier removed ops burden. Strong for production RAG at scale.

Vector extension for Postgres. Eliminates a separate vector DB for moderate scale. Run inside Neon for easy setup.

Structured output library wrapping OpenAI/Anthropic. Pydantic-native, retry logic built-in. Our recommended way to do structured LLM output.

Open-source LLM ops: tracing, evals, prompt management. Self-hostable. Our recommended observability layer for production AI.

AI-assisted coding tools. Proven productivity multipliers for experienced engineers. Copilot's Chat + Workspace features are approaching Cursor quality.

High-performance open-source vector DB. Self-hosted or managed cloud. Strong on filtering and payload indexing.

LLM observability and cost tracking proxy. Easy to integrate, good dashboards. Evaluate against self-hosted options at scale.

Experiment tracking and model eval platform. Strong if you're doing fine-tuning or systematic eval runs.

Unified API proxy across 100+ LLM providers. Useful for multi-provider routing and cost optimization.

Dominant language for AI engineering. Best ecosystem for model inference, ML tooling, and data pipelines. No credible alternative.

For frontend and edge AI features — streaming UI, tool use in the browser, Vercel AI SDK. TypeScript is the right choice here.

Emerging for high-performance inference servers (candle, burn). Not yet ergonomic enough for most teams. Watch this space.

Good for high-throughput proxy / gateway services around LLMs. Not ideal for ML itself but competitive for infrastructure layers.