Tag

vLLM

3 articles tagged with “vLLM”

A/B-testing a-star A2A ACR adaptive-thinking agent-observability agent-patterns agent-protocols agent-security agent-skills agents AGENTS.md aggregation ai ai-gateway AI-Search aks alerting Alertmanager algorithms Anthropic Anthropic-API APE api-design api-gateway API-providers api-versioning apim APM app-router app-service appsec architecture Arize arm-templates ASP.NET-Core aspnet-core aspnetcore assertion-functions asserts async audit authentication authorization AutoGen automation azure Azure-APIM azure-content-safety azure-front-door azure-functions Azure-OpenAI azure-policy azure-storage batch-API bdd beam search Bedrock benchmarks best-practices bfs bi-encoder bias-detection bicep binary-search branching bst Bundler cache-components caching cap-theorem capability-tagging Capitalize cascading case-based-reasoning chain-of-thought charts chatbot checkpointing chunking CI ci-cd ci-optimization cicd class-decorator Claude Claude-Code CLAUDE.md clean-architecture cloud cloud-security cloudevents code-review Codex coding-agents compaction comparison compilerOptions compliance compose concurrency concurrent-rendering conditional-types consistency consistent-hashing constrained-generation constraints consumer-driven container-registry containers content-moderation content-safety context context-compression context-engineering context-window contract-testing conversational-agents Copilot cosmos-db cost-allocation cost-control cost-optimization cost-per-task cost-performance CoT coverage cqrs cross-session csharp Cursor cve cypress .d.ts dapper dashboards dast data-fetching databases dataloader datasets ddd decision-framework declaration-files decorators DeepEval DefinitelyTyped dependency-injection dependency-scanning deployment-slots deployments deprecation design-patterns devops dfs dijkstra discriminated-unions distributed-tracing distributive docker Docker-Hub documentation DORA dotnet dotnet-9 drift-detection DSPy durable-execution dynamic-discovery dynamic-programming e2e-testing edge AI ef-core ef-core-9 embeddings engineering environments episodic-memory error-budget evals evaluation event-driven event-grid event-hubs eventing examples experience-replay experimentalDecorators factuality fail-fast faithfulness feature-flags federation few-shot fiber fine-tuning FinOps flaky-tests flex-consumption frontend function-calling garak Gemini Gemma gen-ai-semantic-conventions generics GHCR git GitFlow GitHub github-actions goal-decomposition golden-signals Google-DeepMind GPT gpt-4o GPT-6 GPU grammar graph graphql Groq grounding GroupChat grpc guardrails hallucination hallucination-detection harmful-content harness-engineering hcl health-checks heap Helm hooks hotchocolate HPA http http2 http3 hybridcache iac ICL idempotency identity image-scanning in-context-learning incident-response indexes infer inference infrastructure input-validation integration integration-testing interoperability introsort invariant isolation Istio jailbreak jest joins jotai JSON JSON-mode jsonb jwt k8s key-remapping key-vault kiota knapsack knowledge-base KQL kubernetes KV cache LangChain Langfuse LangGraph LangSmith language latency least-agency legacy-code Linkerd linq Llama llama.cpp LLM LLM-as-judge LLM-monitoring LLM-security load-balancing logging long-running-agents long-term-memory lost-in-the-middle lower-bound managed-identity mapped-types materialized-views MCP memoization memory memory-consolidation merge message-queue messaging meta-prompting metadata metrics microservices microsoft minimal-apis Mistral mocking mocks Model-Context-Protocol model-routing model-selection model-serving moderation module-augmentation module-resolution modules mongodb monitoring monorepo msw MTEB mTLS multi-agent multi-stage-builds multi-tenant multimodal mvcc narrowing network-security nextjs-16 Node16 normalization nosql oauth oauth2 observability offline-evals oidc Omit on-call on-device online-evals opa open-source open-weight OpenAI OpenAI-API openapi OpenTelemetry OPRO orchestration OTEL Outlines output-format output-validation OWASP P99 paas package-exports pact pactflow page-objects PagerDuty parsing Partial partitioning paths pattern-matching performance personal-AI pgvector Phi Phoenix Pick PII PII-filtering pinecone pipeline planning playwright plugins policy-as-code polly postgresql pricing production production-monitoring progressive-disclosure project-references prompt-caching prompt-design prompt-engineering prompt-injection prompt optimisation promptfoo protobuf pulumi python Qdrant quality quality-gates quality-score quantization queues RAG RAG-evaluation RAGAS rate-limiting react react-19 react-compiler react-internals react-query react-server-components realtime reasoning reasoning-effort Record recursion recursive-types red-black red-green-refactor red teaming redis reflection regression-testing release-management replanning resource-limits Responses-API rest reusable-workflows rls rolling-updates runtime-safety saga-pattern sandboxing sast sbom scale schema schema-design schema-validation SDK sdlc search secrets-management security security-testing self-consistency self-hosting Semantic-Kernel semantic-similarity serializable server-actions serverless service-bus service-mesh sharding shortest-path sidecar-proxy signalr SKILL.md SLI SLM SLO slopsquatting snapshot-testing software-design sorting spec-driven-development speculative-decoding spies sql stability state-graph state-management streaming streams strict-mode stride string-types structured-logging structured-output stubs summarisation supertest supply-chain system-design tail-call tanstack-query task-framing TC39 tdd template-literals templates terraform test-architecture test-doubles test-pyramid testcontainers testing testing-patterns testing-strategy threat-modeling token-bucket token-budget token-economics token-limit token-masking token-tracking tokens tokens-per-second tool-calling tool-registry tool-use topological-sort ToT toxicity tracing trajectory transactions tree-of-thought trees trie trunk-based tsconfig TTFT type-guards type-narrowing type-predicates type-safe-routes type-system type-transforms types typescript unit-testing usage-metrics user-feedback utility-types validation valkey values vault vector-db vector-search vector-store vendor-neutral verification version-control versioning Vertex-AI visual-regression vitest vLLM voice-agents vulnerabilities weaviate web-api web-security WebLLM websockets window-functions Windsurf wiremock workflows working-memory workload-identity workspaces xunit yarp zero-shot zero-trust zod zustand

LLM Model Landscape

Self-hosting an open-weight model: weights cached on a persistent volume load into vLLM on a GPU node pool, a gateway adds authentication and routing, and the deployment scales on queue depth and KV cache usage from Prometheus, with GPU memory split between weights and KV cache.

Self-Hosting LLMs: Running a Model as a Service

AdvancedPRO

Break-even maths, GPU sizing from weights and KV cache, choosing between vLLM, SGLang and llama.cpp, and what to scale and alert on once open-weight models carry production traffic.

06 Oct 20269 min

Read →

LLM Model Landscape

Llama 3.3 70B weights are quantised to fit GPU memory, served with Ollama for development or vLLM for production, behind an OpenAI-style API.

Llama: Open-Weight Models for Production

Intermediate

The Llama lineup and licence, serving with Ollama and vLLM, GPU memory sizing, and an honest look at when a hosted API is still the better choice.

13 OctUpdated 30 Oct 202611 min

Read →

AI Engineering

Requests reach vLLM, where prefix caching skips shared prompt blocks, continuous batching schedules one decoding step at a time, and paged KV cache blocks fill GPU memory, which sets how many requests run at once.

Model Serving With vLLM: The KV Cache Decides Your Capacity

Advanced

KV cache arithmetic, PagedAttention, continuous batching, prefix caching, FP8 and speculative decoding, the vLLM flags that matter, and the five metrics that show which limit you hit.

07 Nov 20268 min

Read →