API Gateways in 2026: YARP, Azure APIM, and the AI Gateway Pattern

Problem Context

An API gateway is the single front door for your APIs: it terminates TLS, authenticates the caller, enforces rate limits, transforms requests, routes to backends, caches responses, and emits the telemetry your ops team actually looks at. Without it, every backend service implements (and gets wrong) the same cross-cutting concerns.

In 2026 the landscape is two tiers. Managed gateways: Azure API Management (now with theAI Gateway feature set — token limits, semantic caching, load balancing across model deployments), AWS API Gateway, Kong Konnect, Apigee. Self-hosted / in-process: YARP(Microsoft's reverse proxy on .NET 9, GA and used inside Azure itself), Envoy, NGINX, Traefik. The pattern of choice for new microservices: YARP for east-west, APIM (or equivalent) for north-south.

Concept Explanation

What the gateway should own:

• Edge security — TLS, JWT/OIDC validation, mTLS to backends, WAF, IP allow-list.
• Traffic shaping — rate limits, quotas, circuit breakers, retries, timeouts.
• Routing — host/path-based routing, blue/green, canary, A/B.
• Transformation — header rewrite, request/response shape, version mapping (v1 in, v2 out).
• Observability — correlation IDs, OpenTelemetry traces, access logs, latency SLOs.
• AI-specific (2026) — token-per-minute caps, semantic cache, load balance across multi-region OpenAI deployments.

What it should notown: business logic. The moment you write an "if order.total > 100" in a gateway policy, you've made the wrong building the system of record.

flowchart LR
    Web["Web client"] --> GW["API Gateway<br/>(APIM / YARP)"]
    Mobile["Mobile"] --> GW
    Partner["Partner B2B"] --> GW
    GW -->|JWT, rate-limit,<br/>transform, log| S1["Orders service"]
    GW --> S2["Catalog service"]
    GW --> S3["Payments service"]
    GW -->|token-limit,<br/>semantic cache| AI["Azure OpenAI<br/>(multi-region)"]

    style GW fill:#0078D4,color:#fff,stroke:#005a9e

Implementation

Step 1: YARP as an in-process reverse proxy (.NET 9)

// dotnet add package Yarp.ReverseProxy

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddReverseProxy()
    .LoadFromConfig(builder.Configuration.GetSection("ReverseProxy"));

var app = builder.Build();
app.MapReverseProxy();
app.Run();

// appsettings.json
// "ReverseProxy": {
//   "Routes": {
//     "orders": { "ClusterId": "orders", "Match": { "Path": "/api/orders/{**catch-all}" } }
//   },
//   "Clusters": {
//     "orders": {
//       "Destinations": { "primary": { "Address": "https://orders.internal/" } },
//       "HttpRequest": { "Version": "2.0" }
//     }
//   }
// }

Step 2: Auth, rate limit, and CORS at the gateway

builder.Services
    .AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
    .AddJwtBearer(o =>
    {
        o.Authority = "https://login.example.com";
        o.Audience  = "api://orders";
    });

builder.Services.AddRateLimiter(o =>
    o.AddTokenBucketLimiter("default", opt =>
    {
        opt.TokenLimit = 200;
        opt.TokensPerPeriod = 200;
        opt.ReplenishmentPeriod = TimeSpan.FromMinutes(1);
    }));

builder.Services.AddCors(o => o.AddDefaultPolicy(p => p
    .WithOrigins("https://app.example.com")
    .AllowAnyHeader().AllowAnyMethod()));

app.UseCors();
app.UseAuthentication();
app.UseRateLimiter();
app.MapReverseProxy().RequireAuthorization().RequireRateLimiting("default");

Step 3: Request transformation in YARP

builder.Services.AddReverseProxy()
    .LoadFromConfig(builder.Configuration.GetSection("ReverseProxy"))
    .AddTransforms(t =>
    {
        // Strip the Authorization header, replace with a backend-trusted token
        t.AddRequestTransform(async ctx =>
        {
            ctx.ProxyRequest.Headers.Remove("Authorization");
            var backendToken = await _tokenSvc.GetBackendTokenAsync(ctx.HttpContext.User);
            ctx.ProxyRequest.Headers.Authorization =
                new AuthenticationHeaderValue("Bearer", backendToken);
        });

        // Inject correlation ID
        t.AddRequestTransform(ctx =>
        {
            ctx.ProxyRequest.Headers.Add("X-Correlation-Id",
                ctx.HttpContext.TraceIdentifier);
            return ValueTask.CompletedTask;
        });
    });

Step 4: Azure APIM policy (north-south traffic)

<!-- inbound -->
<validate-jwt header-name="Authorization" failed-validation-httpcode="401">
  <openid-config url="https://login.example.com/.well-known/openid-configuration" />
  <required-claims>
    <claim name="aud"><value>api://orders</value></claim>
  </required-claims>
</validate-jwt>

<rate-limit-by-key calls="1000" renewal-period="60"
                   counter-key="@(context.Subscription.Id)" />

<set-header name="X-Forwarded-Tenant" exists-action="override">
  <value>@(context.User.Groups.GetValueOrDefault("tenant_id"))</value>
</set-header>

<!-- outbound -->
<set-header name="X-Powered-By" exists-action="delete" />
<cache-store duration="60" />

Step 5: AI Gateway pattern (token caps + semantic cache)

<!-- APIM AI Gateway policies for Azure OpenAI backends -->
<azure-openai-token-limit
    tokens-per-minute="50000"
    counter-key="@(context.Subscription.Id)"
    estimate-prompt-tokens="true" />

<azure-openai-emit-token-metric                           <!-- to App Insights -->
    namespace="openai">
  <dimension name="API ID" />
  <dimension name="Subscription ID" />
</azure-openai-emit-token-metric>

<azure-openai-semantic-cache-lookup
    score-threshold="0.05"
    embeddings-backend-id="embeddings-deployment" />
<!-- Cuts cost 30-60% for repetitive prompts; caps runaway spend per tenant. -->

Step 6: Health checks, circuit breaking, and OTel

// YARP active + passive health checks per cluster
// "HealthCheck": {
//   "Active":  { "Enabled": true, "Path": "/healthz", "Interval": "00:00:10" },
//   "Passive": { "Enabled": true, "ReactivationPeriod": "00:00:30" }
// }

// Circuit-style retry policy via SocketsHttpHandler / Polly
builder.Services.AddReverseProxy()
    .ConfigureHttpClient((ctx, handler) =>
    {
        handler.PooledConnectionLifetime = TimeSpan.FromMinutes(2);
    });

// OpenTelemetry — gateway is the perfect place to start the trace
builder.Services.AddOpenTelemetry()
    .WithTracing(t => t.AddAspNetCoreInstrumentation()
                       .AddHttpClientInstrumentation()
                       .AddOtlpExporter());

Common Pitfalls

1. Business logic in gateway policies. Routing, auth, transformation — yes. Pricing rules, workflow state, validation beyond shape — no. The gateway becomes the second source of truth and nothing is testable.
2. One giant gateway for everyone. Mobile, web, and partners have different needs. UseBackend-for-Frontend (BFF): a small gateway per channel.
3. Skipping auth at the gateway because "the service does it." Defense in depth. Validate the JWT at the edge and again at the service. Costs almost nothing; catches misconfiguration.
4. No timeout / no circuit breaker. One slow downstream service hangs every gateway worker. Set per-cluster timeouts (1-5s typical) and circuit-break on consecutive failures.
5. Logging request bodies by default. Compliance nightmare. Log shape and size, not content. Mask known PII fields.
6. For AI workloads: no token caps. A buggy retry loop or a prompt injection can burn $1,000s overnight against GPT-class endpoints. Token-per-minute limits at the gateway are non-optional in 2026.

Practical Takeaways

• Gateway owns: TLS, auth, rate limit, transform, route, observe. Nothing else.
• YARP for in-process / east-west on .NET 9. APIM for north-south at the org boundary.
• BFF pattern: one gateway per channel (mobile, web, partner) beats one giant gateway.
• Defense in depth: validate JWT at the gateway and at the service.
• Set timeouts and circuit breakers per-cluster. Always.
• For AI backends: token-per-minute caps + semantic cache + token metrics. Save money, prevent surprise bills.
• Start the OpenTelemetry trace at the gateway — it's the only place that sees every request.