Works with the AI Agents you already use
llm inference optimization
Systematic optimization for LLM latency, throughput, and memory usage across major serving frameworks.
$5
/moSubscription · cancel anytime
Secure checkout via Stripe
- Always the latest version
- Delivered live to your agent via MCP
- Cancel anytime, access ends at period end
llm inference optimization
Example session with this skill installed
My Llama-3-70B instance is hitting OOM on an A100 and the latency is too high for real-time chat. How do I optimize this for vLLM?
- Read your context and instructions
- Compiled the llm inference optimization
I have analyzed your deployment constraints for Llama-3-70B on an A100. To resolve the Out-of-Memory (OOM) errors and reduce latency, you should implement a combination of 4-bit quantization and PagedAttention optimizations.
Optimization Strategy for Llama-3-70B
| Component | Recommendation | Impact |
|---|---|---|
| Quantization | AWQ 4-bit (W4A16) | ~50% VRAM reduction |
| Serving Framework | vLLM with PagedAttention | Eliminates memory fragmentation |
| KV Cache | FP8 KV Cache Scaling | Reduces memory footprint of long contexts |
| Batching | Continuous Batching | Increases throughput under load |
| Attention | FlashAttention-2 | Optimizes prefill and decode speed |
Recommended Deployment Command
vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
--quantization awq \
--dtype half \
--gpu-memory-utilization 0.95 \
--max-model-len 4096 \
--enable-prefix-caching \
--kv-cache-dtype fp8
Next steps
- Benchmark TTFT and TPOT using the vLLM benchmark suite to ensure the AWQ quantization meets your quality thresholds.
- Monitor GPU memory fragmentation via the vLLM metrics endpoint to tune
gpu-memory-utilization. - Enable speculative decoding with a Llama-3-8B draft model if TPOT remains above 50ms.
Subscription · always the latest version
- Secure checkout via Stripe
- Cancel anytime
- Delivered live via MCP
About this skill
High latency and memory bottlenecks stall LLM deployments in production. This skill provides a systematic workflow to reduce Time to First Token (TTFT), increase tokens per second, and minimize GPU memory consumption.
What it does
- Performance profiling identifies bottlenecks in TTFT, TPOT, and memory usage before applying optimizations.
- Quantization selection guides the implementation of FP8, AWQ, or GPTQ based on hardware constraints and quality requirements.
- Serving stack optimization matches workloads to frameworks like vLLM, SGLang, or TensorRT-LLM for maximum throughput.
- KV cache management implements PagedAttention and prefix caching to eliminate memory fragmentation.
- Decoding acceleration uses speculative decoding and FlashAttention kernels to speed up token generation.
How it works
- Analyze metrics using the profiling step to establish baselines for latency and GPU utilization.
- Select framework and quantization levels based on the specific model size and available hardware.
- Configure server parameters for batching, scheduling, and cache reuse to handle concurrent requests.
Frameworks & tools
Supports vLLM, SGLang, TensorRT-LLM, HuggingFace TGI, and llama.cpp. Compatible with NVIDIA Hopper architectures for FP8 and various quantization methods including AWQ and GPTQ.
Why this beats prompting it yourself
Generic prompts often miss hardware-specific optimizations like FlashDecoding or prefix-aware batching. This skill provides a structured decision tree that balances the trade-offs between model quality, speed, and cost without blind trial and error.
Use cases
- Reducing the serving cost of 70B+ parameter models on limited VRAM.
- Optimizing agentic workflows that rely on heavy system prompt reuse.
- Improving the responsiveness of real-time chat applications by lowering TTFT.
- Scaling throughput for high-volume batch processing tasks.
Known limitations
Optimization results depend heavily on specific GPU architecture. Higher levels of quantization like W4A4 may result in noticeable model quality degradation.
Hardware dependent. Results vary by GPU architecture. Aggressive quantization (e.g., W4A4) may degrade output quality. Requires initial profiling to be effective.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Free forever
No account required to browse
Trust & safety
Security scan
Security scan passed 22 days ago