Designing RAG Systems That Actually Scale

Problem Context

Every team building with LLMs eventually arrives at the same conclusion: the model doesn't know your data. Retrieval-Augmented Generation (RAG) is the standard answer — ground the model in your documents, databases, or knowledge base at query time.

The problem? Most RAG tutorials stop at "put your PDFs in a vector store and call it done." In production, that approach breaks at every seam: irrelevant retrieval, stale embeddings, latency spikes, and hallucinations that look grounded but aren't.

Concept Explanation

A production RAG system has four critical stages, each with its own failure modes:

      flowchart LR
          A["Document Ingestion"] --> B["Chunking and Preprocessing"]
          B --> C["Embedding and Indexing"]
          C --> D["Retrieval and Ranking"]
          D --> E["LLM Generation"]
          E --> F["Response + Citations"]
      
          style A fill:#4f46e5,color:#fff,stroke:#4338ca
          style B fill:#4f46e5,color:#fff,stroke:#4338ca
          style C fill:#7c3aed,color:#fff,stroke:#6d28d9
          style D fill:#7c3aed,color:#fff,stroke:#6d28d9
          style E fill:#059669,color:#fff,stroke:#047857
          style F fill:#059669,color:#fff,stroke:#047857
      

Stage 1: Chunking Strategies

Chunking is where most RAG systems silently fail. The wrong chunk size means either too little context (the LLM can't synthesize an answer) or too much noise (irrelevant content dilutes the signal).

Fixed-size chunking (e.g., 512 tokens with 50-token overlap) is the baseline. It's fast and predictable but ignores document structure entirely. A code sample split mid-function is useless.

Semantic chunking splits on natural boundaries — paragraphs, sections, or topic shifts detected via embedding similarity. This preserves meaning but adds preprocessing cost.

Recursive chunking first splits by structure (headings, code blocks), then subdivides large sections by token count. This is the sweet spot for most production systems.

# Recursive chunking with LangChain
      from langchain.text_splitter import RecursiveCharacterTextSplitter
      
      splitter = RecursiveCharacterTextSplitter(
          chunk_size=800,
          chunk_overlap=100,
          separators=["\n## ", "\n### ", "\n\n", "\n", ". ", " "],
          length_function=len,
      )
      
      chunks = splitter.split_documents(documents)
      

Stage 2: Embedding Pipeline

Your embedding model is the bottleneck you don't see. Key decisions:

• Model choice: text-embedding-3-large (Azure OpenAI) for general use. Cohere's embed-v3 if you need multilingual. Don't use ada-002 for new projects — it's a generation behind.
• Dimensionality: 1536 dims is standard. Azure OpenAI's large model supports dimensions parameter to reduce to 256 or 512 with minimal quality loss — cuts storage and search cost significantly.
• Batch processing: Embed in batches of 100–500. Single-item embedding calls will destroy your throughput and cost budget.

// Azure OpenAI embedding with reduced dimensions
      var embeddingOptions = new EmbeddingGenerationOptions
      {
          Dimensions = 512  // Reduce from 3072 default
      };
      
      var response = await client.GenerateEmbeddingsAsync(
          new[] { "Your chunk text here" },
          embeddingOptions
      );
      

Stage 3: Retrieval Patterns

Vector similarity alone is often insufficient. Production systems layer multiple retrieval strategies:

Hybrid search combines vector similarity with keyword (BM25) search. Azure AI Search does this natively with search +vectorQueries in a single API call. This catches exact-match terms that embedding similarity misses.

Re-ranking applies a cross-encoder model to the top-K results from initial retrieval. This is computationally expensive per-item but dramatically improves precision when applied to 20–50 candidates. Azure AI Search offers built-in semantic ranking for this.

Query expansion rewrites the user query into multiple search variants using the LLM itself, then merges results. This handles ambiguous queries where the user's phrasing doesn't match document terminology.

Stage 4: Generation with Grounding

Pass retrieved chunks as context with explicit grounding instructions:

System: Answer the user's question using ONLY the provided context.
      If the context doesn't contain sufficient information, say so.
      Cite sources using [1], [2] notation.
      
      Context:
      [1] {chunk_1_text} (source: design-doc.md, section: Architecture)
      [2] {chunk_2_text} (source: api-spec.yaml, section: Endpoints)
      
      User: {query}
      

Implementation

A scalable RAG pipeline on Azure looks like this:

      flowchart TD
          subgraph Ingestion
              A["Blob Storage"] --> B["Azure Function Trigger"]
              B --> C["Document Cracking + Chunking"]
              C --> D["Azure OpenAI Embeddings"]
              D --> E["Azure AI Search Index"]
          end
      
          subgraph Query
              F["User Query"] --> G["Query Embedding"]
              G --> H["Hybrid Search + Semantic Ranking"]
              H --> I["Top-K Chunks"]
              I --> J["Azure OpenAI GPT-4o"]
              J --> K["Grounded Response"]
          end
      
          E -.->|vector index| H
      
          style A fill:#d97706,color:#fff,stroke:#b45309
          style E fill:#4f46e5,color:#fff,stroke:#4338ca
          style J fill:#059669,color:#fff,stroke:#047857
          style K fill:#059669,color:#fff,stroke:#047857
      

Key implementation details:

• Ingestion: Trigger on blob upload via Event Grid. Use Azure Document Intelligence for PDFs, images, and scanned docs — it extracts structure (tables, headings) that plain text extraction misses.
• Indexing: Azure AI Search with vector + full-text fields. Configure vectorSearch with HNSW algorithm,efConstruction: 400 for index build quality.
• Query path: Always call embedding API + search API in parallel where possible. Total retrieval should be under 200ms.

// Hybrid search with Azure AI Search
      var searchOptions = new SearchOptions
      {
          Size = 10,
          Select = { "content", "title", "source", "chunk_id" },
          QueryType = SearchQueryType.Semantic,
          SemanticSearch = new SemanticSearchOptions
          {
              SemanticConfigurationName = "default",
              QueryCaption = new QueryCaption(QueryCaptionType.Extractive)
          },
          VectorSearch = new VectorSearchOptions
          {
              Queries =
              {
                  new VectorizedQuery(queryEmbedding)
                  {
                      KNearestNeighborsCount = 20,
                      Fields = { "contentVector" }
                  }
              }
          }
      };
      
      var results = await searchClient.SearchAsync<SearchDocument>(
          userQuery,  // BM25 keyword search
          searchOptions  // + vector search + semantic ranking
      );
      

Pitfalls

Practical Takeaways