LLM API Providers Compared

The API Call Is the Easy Part

Every LLM provider offers a POST /chat/completions-shaped endpoint, a Python SDK, and a pricing page. The hard part is everything around the call. Which provider your compliance team will actually sign off on. Whether your rate limit survives a traffic spike. What happens when a vendor changes its content policy under you without much warning. "Just call the API" is a one-week decision that quietly turns into a multi-year platform commitment, and most teams don't notice until the second provider shows up.

By 2026 the landscape has settled into three genuinely different categories. Not three tiers of the same thing, three different answers to "where does this call actually go." Treating them as interchangeable is where most provider decisions go sideways.

Three Categories, Not One Spectrum

Direct vendor APIs — OpenAI, Anthropic, Google — give you the newest models on day one, plus the vendor's own billing console and rate limit dashboard. Cloud-hosted offerings (Azure OpenAI, AWS Bedrock, Vertex AI) put the same or similar models behind your existing cloud account: your IAM, your VPC, your billing relationship. In exchange you usually wait days to weeks after a model launches for it to show up on the cloud platform, and you pick up a thin platform layer between you and the model (deployment names, quota, region availability) that the direct API doesn't have. Inference-as-a-service providers are a different animal entirely: Together AI, Groq, Fireworks don't train anything. They host open-weight models — Llama, the Mistral/Mixtral family, and others — on infrastructure built for throughput, and they can undercut proprietary frontier pricing by a wide margin because the underlying weights are free.

flowchart TD
    Need["You need an LLM API"] --> Q{"What matters most?"}
    Q -->|"Compliance, data residency,
existing cloud commitment"| Cloud["Cloud-hosted
Azure OpenAI / Bedrock / Vertex AI"]
    Q -->|"Newest models, simplest
integration, vendor-native tooling"| Direct["Direct vendor
OpenAI / Anthropic / Google"]
    Q -->|"Cheapest, fastest inference
on open-weight models"| Inference["Inference-as-a-service
Together / Groq / Fireworks"]

    style Need fill:#4f46e5,color:#fff,stroke:#4338ca
    style Direct fill:#059669,color:#fff,stroke:#047857
    style Cloud fill:#d97706,color:#fff,stroke:#b45309
    style Inference fill:#7c3aed,color:#fff,stroke:#6d28d9
      

Why Enterprises Route Through Azure OpenAI or Bedrock

Teams that could call OpenAI or Anthropic directly often don't, and it's rarely a technical call. If your company already has an Enterprise Agreement with Microsoft or an EDP with AWS, adding an AI workload is a line item against a budget that already exists, not a new vendor relationship with its own DPA and its own security review starting from zero. Data residency is the other big one: Azure OpenAI and Bedrock let you pin inference to a region and keep traffic inside a VPC boundary that has already passed audit, which matters a lot more to a regulated bank than it does to a five-person startup. And because committed cloud spend applies to AI usage, the marginal cost of a new feature often looks like it's coming out of a budget line that was going to be spent anyway. None of this makes the model better. It makes the procurement conversation shorter, which in a large org is sometimes the harder problem.

The trade-off is real and worth saying plainly: cloud-hosted deployments lag the direct API on new model availability, sometimes by weeks, and you inherit a platform layer — deployment names, provisioned throughput units, per-region quota — that the direct API simply doesn't have. For a startup with no compliance overhead, that trade-off usually isn't worth it. Go direct and skip the extra layer.

Rate Limits and Usage Tiers

Every major provider gates you into a usage tier based on account age and spend, then raises your requests-per-minute and tokens-per-minute ceilings as you spend more. This is invisible right up until it isn't — a demo that ran fine at ten requests a minute can start throwing 429s the moment a feature ships to real traffic, and the fix (asking the vendor for a tier bump) is not instant. Azure OpenAI makes this worse in one specific way: quota is provisioned per deployment and per region, so a launch can be blocked by a quota request that should have been filed a week earlier, not the day before.

// Minimal exponential backoff for 429s — works across providers,
// since all of them return a 429 with a Retry-After-shaped signal.
async function callWithBackoff<T>(
  fn: () => Promise<T>,
  maxRetries = 5
): Promise<T> {
  let attempt = 0;
  while (true) {
    try {
      return await fn();
    } catch (err: any) {
      const status = err?.status ?? err?.response?.status;
      if (status !== 429 || attempt >= maxRetries) throw err;

      const retryAfterHeader = err?.response?.headers?.['retry-after'];
      const waitMs = retryAfterHeader
        ? Number(retryAfterHeader) * 1000
        : Math.min(30_000, 500 * 2 ** attempt) + Math.random() * 250;

      await new Promise((r) => setTimeout(r, waitMs));
      attempt++;
    }
  }
}

The Multi-Provider Abstraction Problem

Plenty of providers, including most inference-as-a-service vendors, expose an "OpenAI-compatible" endpoint: same request shape, same chat.completions.createcall. It looks like a drop-in replacement right up until you need something provider-specific. Anthropic puts the system prompt in its own top-level field instead of a message role. Prompt caching syntax differs per vendor. Tool-call argument validation is stricter on some platforms than others. A content-filter refusal comes back in a shape that doesn't map cleanly to OpenAI's error format, so your error handling silently breaks the moment you point the same code at a second provider. Compatibility gets you most of the way there and then leaks at exactly the moment you're relying on it most.

That leak is why most teams running more than one model in production build a thin gateway layer. Not a full abstraction that hides every provider difference — that tends to become a maintenance tax of its own — but a routing and normalization layer that keeps application code provider-agnostic for the common path while still letting provider-specific features through when you need them.

interface LlmGateway {
  complete(req: NormalizedRequest, opts?: { provider?: string }): Promise<NormalizedResponse>;
}

// Route by task requirement, not by hardcoding a vendor name everywhere
class ProviderRouter implements LlmGateway {
  constructor(private adapters: Record<string, LlmGateway>) {}

  async complete(req: NormalizedRequest, opts?: { provider?: string }) {
    const provider = opts?.provider ?? this.selectProvider(req);
    return this.adapters[provider].complete(req);
  }

  private selectProvider(req: NormalizedRequest): string {
    if (req.requiresDataResidency) return 'azure-openai';
    if (req.inputTokens > 500_000) return 'gemini';   // needs huge context
    if (req.costSensitive) return 'together';          // open-weight, cheap
    return 'anthropic';                                 // default
  }
}

Decision Framework

Key Takeaways