llm gpu fit

The problem

Running local LLMs often leads to OOM crashes or empty agent responses because standard VRAM math ignores KV cache growth and architectural differences. You waste hours downloading 10GB models only to find they lack the context window your agent requires or hide reasoning in non-standard fields.

What it does

• Performs architecture-aware VRAM estimation by calculating KV cache costs for MHA, GQA, and MQA models.
• Conducts a 3-way fit check verifying VRAM headroom, context window minimums, and agent field compatibility.
• Identifies models that break OpenAI-compatible tools by routing output to "thinking" fields instead of the standard content field.
• Provides precise curl commands to verify model loading and create custom context length variants without re-downloading weights.

Frameworks & tools

Ollama, NVIDIA SMI, CUDA, and OpenAI-compatible inference servers.

Why this beats prompting it yourself

Generic LLMs often use "paper math" that underestimates memory usage by ignoring the tokeniser and CUDA context overhead. This skill uses empirical data from real hardware testing to account for the specific multipliers in modern architectures like Qwen and Llama.

Use cases

• Pre-flighting a model selection for a local agent requiring a 64K+ context window.
• Debugging why a model "fits" on paper but fails to return content to your application.
• Optimizing GPU utilization by switching from MHA to GQA models for long-context tasks.
• Automating the creation of model variants with specific num_ctx parameters.

Known limitations

Estimates are based on Q4_K_M quantization as the baseline. Real VRAM usage may vary slightly based on specific driver versions and system background tasks.