- Home
- Skills
- DevOps & Cloud
- vLLM: High-Throughput LLM Serving
More screenshots
Works with the AI Agents you already use
vLLM: High-Throughput LLM Serving
Serve open models in production with vLLM: paper-first VRAM capacity math, a four-rung deployment ladder, online vs. offline modes, quantization checks, and load-tested limits.
$5
vLLM: High-Throughput LLM Serving
Example session with this skill installed
We need to serve a 32B open model for an internal document-QA tool. About 40 engineers, documents up to roughly 30k tokens, and we have two 48GB GPUs. Latency should feel responsive but this is internal, not customer-facing. Plan the deployment.
- Read your context and instructions
- Compiled the vllm
Capacity math first. At FP16 the weights alone are 32B × 2 bytes = 64GB. Two 48GB GPUs give 96GB total, so with tensor parallelism across both cards the weights fit with about 32GB left for KV cache and overhead — workable, but tight for 30k-token contexts at real concurrency. The alternative worth pricing before you commit: an AWQ 4-bit checkpoint lands near 18GB, fits on one GPU, and leaves the second card free — at the cost of a quantization quality check you must actually run. Given the 30k context requirement, the plan takes the FP16 two-GPU route and keeps AWQ as the fallback if the load test shows KV pressure you cannot afford.
Configuration, by the ladder: rung 1 sanity check for coherent output with the right chat template; then tensor-parallel size 2, GPU memory utilization 0.9, and max model length set to 32768 — your real document size, not the model's maximum. Verify both GPUs show utilization during a request; an idle second card means the parallelism flag did not take.
Load test with real documents, including the 30k-token worst case, ramping concurrency in steps. Expect the knee earlier than intuition says — long contexts make KV cache, not weights, the binding constraint. Set the concurrency cap below the measured knee (expect somewhere in the 8–16 range, but the test decides, not this plan), and define over-cap behavior as short queueing, then a fast, clear busy error.
Production rung: containerized, request timeout 300 seconds for the longest documents, metrics endpoint on, health check wired. Document the cap and the fallback: if KV pressure bites in week one, the pre-approved moves are capping context at 24k or switching to the verified AWQ checkpoint — decided now, not during an incident.
Connects securely to your tools. The creator never sees your data.
About this skill
A model that runs in a notebook is not a model that serves. Production serving fails in predictable places: nobody did the VRAM math, a laptop configuration got deployed as a production one, the context window was set to the brochure maximum and ate the KV cache, or the first real traffic found the concurrency knee that a load test would have found first. This skill is a runbook that prevents those failures in order. It starts with an inputs/outputs contract and a stop rule — a single-user experiment does not need a server. Step one is capacity math on paper: weights at your precision, KV cache growth with context and concurrency, and a hard headroom rule, captured in a fill-in capacity worksheet. Step two climbs a four-rung deployment ladder — local sanity, configured single GPU, verified multi-GPU, then production with timeouts, metrics, health checks, and a container — with an explicit warning against skipping rungs. Step three chooses the serving mode with a comparison table: online API serving tuned for latency, or offline batch inference through the in-process class when you have a prompt list and a deadline instead of users. Step four covers quantization strategy (AWQ/GPTQ 4-bit, FP8) with the non-negotiable rule that a quantized checkpoint gets verified against the full-precision baseline on your own evals. Step five is load-test discipline: representative prompts including your longest real case, a gradual concurrency ramp, and the max-concurrency cap set below the measured knee. A complete worked example (an 8B model served to a team on one 24GB GPU), a pre-launch checklist, a five-dimension readiness rubric, an extended troubleshooting table, and edge cases from giant-prompt stalls to context creep close it out.
- It plans and validates deployments; it does not provision servers or run the load test for you.
- Capacity figures are planning estimates — your measured load test is the arbiter, and the skill says so.
- It is vLLM-specific in its configuration guidance; the math and discipline transfer to other engines, the flags do not.
- It covers single-service deployments, not multi-region routing, autoscaling policy, or model selection.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 24 days ago
Permissions
No special permissions declared or detected