Small Language Models in Production

Problem Context

GPT-4o is amazing. It's also $2.50 per million input tokens and adds 500ms+ latency to every request. When you're classifying support tickets, extracting entities from logs, or routing user intents — tasks that don't need the world's most capable model — you're paying a premium for capability you don't use.

Small language models (SLMs) — models with 1-8 billion parameters like Phi-3, Gemma 2, Mistral 7B, and Llama 3.1 8B — can handle many production tasks at a fraction of the cost and latency. But they fail differently than large models, and knowing where the quality cliff is matters as much as knowing where they excel.

Concept Explanation

Small models aren't "dumb big models." They're trained with different strategies — distillation, synthetic data, domain focus — that give them strong performance on specific task types. The key insight: model size determines the ceiling of reasoning complexity, not thequality on well-scoped tasks.

      flowchart LR
          T["Task Complexity"] --> S{"Scope?"}
          S -->|"Narrow + Well-defined"| SM["Small Model\n(1-8B params)"]
          S -->|"Open-ended + Reasoning"| LM["Large Model\n(70B+ or API)"]
          SM --> D["Deploy: Local / Azure\nLatency: 20-80ms\nCost: ~$0.01/1M tokens"]
          LM --> API["Deploy: API\nLatency: 200-800ms\nCost: $2-15/1M tokens"]
      
          style SM fill:#059669,color:#fff,stroke:#047857
          style LM fill:#4f46e5,color:#fff,stroke:#4338ca
          style D fill:#059669,color:#fff,stroke:#047857
          style API fill:#4f46e5,color:#fff,stroke:#4338ca
      

Where Small Models Excel

• Classification: Sentiment, intent routing, topic categorization, spam detection
• Entity extraction: Names, dates, amounts, product codes from structured text
• Summarization: Short text summaries with clear constraints
• Code tasks: Autocompletion, simple refactoring, syntax transformation
• Translation: Common language pairs with limited domain

Where Small Models Struggle

• Multi-step reasoning: Tasks requiring 3+ logical hops
• Complex instruction following: Many constraints simultaneously
• Novel problem solving: Tasks unlike anything in training data
• Long context: Performance degrades significantly past 8K tokens

Implementation

Step 1: Model Selection for Common Tasks

// Model routing based on task complexity
      public class ModelRouter
      {
          private readonly IChatClient _smallModel;   // Phi-3-mini via ONNX
          private readonly IChatClient _largeModel;   // GPT-4o via Azure OpenAI
      
          public IChatClient Route(TaskType task) => task switch
          {
              TaskType.Classification => _smallModel,
              TaskType.EntityExtraction => _smallModel,
              TaskType.IntentRouting => _smallModel,
              TaskType.ShortSummary => _smallModel,
              TaskType.CodeCompletion => _smallModel,
      
              TaskType.ComplexReasoning => _largeModel,
              TaskType.LongDocumentAnalysis => _largeModel,
              TaskType.CreativeGeneration => _largeModel,
              TaskType.MultiStepPlanning => _largeModel,
      
              _ => _largeModel // Default to large for unknown tasks
          };
      }
      

Step 2: Running Phi-3 with ONNX Runtime

// Local inference with ONNX Runtime GenAI
      using Microsoft.ML.OnnxRuntimeGenAI;
      
      var modelPath = "models/phi-3-mini-4k-instruct-onnx";
      using var model = new Model(modelPath);
      using var tokenizer = new Tokenizer(model);
      
      var prompt = $"""
          <|system|>Classify the support ticket into: billing, technical, account, other.
          Return only the category name.<|end|>
          <|user|>{ticketText}<|end|>
          <|assistant|>
          """;
      
      var sequences = tokenizer.Encode(prompt);
      using var genParams = new GeneratorParams(model);
      genParams.SetSearchOption("max_length", 20);
      genParams.SetSearchOption("temperature", 0.1);
      genParams.SetInputSequences(sequences);
      
      using var generator = new Generator(model, genParams);
      // Generate tokens
      while (!generator.IsDone())
          generator.ComputeLogits();
          generator.GenerateNextToken();
      
      var output = tokenizer.Decode(generator.GetSequence(0));
      

Step 3: Deploying on Azure with Serverless Pay-Per-Token

// Azure AI Model Inference — works with deployed SLMs
      var client = new ChatCompletionsClient(
          new Uri("https://your-endpoint.inference.ai.azure.com"),
          new AzureKeyCredential(apiKey)
      );
      
      var response = await client.CompleteAsync(new ChatCompletionsOptions
      {
          Messages =
          {
              new ChatRequestSystemMessage("Classify into: billing, technical, account, other."),
              new ChatRequestUserMessage(ticketText)
          },
          Temperature = 0.1f,
          MaxTokens = 10
      });
      

Step 4: Quality Gate — When to Escalate

public class QualityGatedRouter
      {
          public async Task<string> ClassifyWithFallback(string input)
          {
              // Try small model first
              var result = await _smallModel.ClassifyAsync(input);
      
              // Check confidence — escalate on low confidence
              if (result.Confidence < 0.85)
              {
                  var fallback = await _largeModel.ClassifyAsync(input);
                  _metrics.RecordEscalation("classification", result.Confidence);
                  return fallback.Category;
              }
      
              return result.Category;
          }
      }
      

Pitfalls

Practical Takeaways