local llm selection

    1

    Technical framework for choosing local LLMs, sizing quantization to VRAM, and weighing QLoRA vs RAG.

    Free

    2 usesSecurity scanned

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    local llm selection

    Example session with this skill installed

    I have a workstation with 12GB VRAM. Should I run a 7B or 14B model for a local coding assistant, and which quantization should I use? Also, should I fine-tune it for my company's style?

    • Read your context and instructions
    • Compiled the local llm selection

    I have analyzed your hardware constraints and requirements for a local coding assistant. Based on your 12GB VRAM limit, a 7B model is the viable choice to ensure stability and sufficient context headroom.

    Local LLM Hardware & Strategy Analysis

    CategoryRecommendationTechnical Details
    Model Size7B - 8B Parameter Class14B models at usable quants exceed 12GB VRAM once KV cache is added.
    QuantizationQ4_K_M or Q5_K_MQ4 (~5GB) or Q5 (~6GB) leaves 5-6GB for context and system overhead.
    Model FamilyQwen-Coder or StarCoder2Specialized code models outperform general chat models at this size.
    StrategyPrompting/RAG over QLoRAStart with RAG. Fine-tune only if style remains inconsistent after prompting.
    DeploymentOllama / llama.cppUse GGUF format for easy quantization management on consumer GPUs.

    VRAM Allocation Estimate

    • Model Weights (Q5_K_M): ~6.0 GB
    • KV Cache (8k context): ~1.5 - 2.0 GB
    • System/GUI Overhead: ~1.0 GB
    • Remaining Headroom: ~3.0 GB (Safe margin for stability)

    Next steps

    • Install Ollama and pull a code-specialized 7B model at Q4 or Q5 quantization.
    • Implement a provider-switch in your application logic to gate this model behind a DEV_OFFLINE=1 flag.
    • Establish a baseline eval set to test the model's output against your company's coding style before considering QLoRA.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Running a local LLM is rarely just about saving money on API credits. It introduces operational overhead, hardware constraints, and potential quality trade-offs that can stall a project. This skill provides a technical framework for evaluating whether to stick with a hosted API or migrate to a local stack like Ollama, llama.cpp, or vLLM. ## What it does - **Local vs hosted evaluation** compares privacy, volume, and latency requirements to determine the best deployment strategy. - **VRAM sizing** calculates model and quantization requirements against hardware limits to prevent out-of-memory errors. - **Quantization selection** weighs the trade-offs between FP16, Q8, Q5, and Q4 formats to balance speed and quality. - **Fine-tuning triage** determines if a project requires QLoRA or if RAG and advanced prompting are sufficient. - **Architecture planning** defines how to implement env-gated provider paths for swappable local/hosted backends. ## How it works 1. **Define the workload requirements** including data privacy needs, expected volume, and hardware availability. 2. **Calculate VRAM overhead** by multiplying parameters by bytes-per-weight and adding headroom for KV cache. 3. **Compare model specialized variants** like choosing a code-tuned model over a general chat model for dev tasks. 4. **Draft a provider-agnostic implementation** using a clean architecture pattern to gate local models behind environment flags. ## Frameworks & tools The skill focuses on local inference engines like Ollama, llama.cpp, and vLLM. It handles quantization formats including GGUF and EXL2, and covers fine-tuning workflows involving QLoRA and PEFT. ## Why this beats prompting it yourself Generic LLMs often provide overly optimistic advice about local hosting or fail to account for the specific VRAM footprint of KV caches. This skill enforces a "hosted-first" production bias and requires rigorous cost-complexity math before recommending hardware-bound solutions. ## Use cases - Deciding if a high-volume data pipeline should migrate from OpenAI to local vLLM nodes. - Selecting the largest possible coding model that fits on a developer's 12GB or 16GB GPU. - Evaluating whether inconsistent JSON output justifies a QLoRA fine-tune vs improved RAG. - Setting up a local development environment that works offline without breaking production API paths. ## Known limitations Does not provide benchmark scores for specific model versions. Hardware calculations assume standard transformer architectures and may not account for specialized NPU or Apple Silicon optimizations.

    Hardware calculations assume standard transformer architectures. Does not provide specific quality benchmarks for individual models. Focuses on technical sizing rather than prompt engineering.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    2 uses

    Times used to date

    Free forever

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 28 days ago

    • Free with an Agensi account

    Permissions

    No special permissions declared or detected

    Listed28 days ago

    What's inside

    Frequently Asked Questions

    This skill provides a decision-making framework to determine the exact hardware requirements, quantization levels, and architectural patterns needed to move AI workloads from cloud APIs to local hardware without performance loss.

    The skill provides logic for sizing models against specific VRAM limits (e.g., 8GB, 12GB, 24GB) and calculating the necessary overhead for KV caches to prevent runtime crashes.

    Yes, the skill includes a 'Fine-tuning Triage' module that helps you weigh the cost and complexity of a QLoRA fine-tune against the speed and simplicity of a RAG-based approach.

    You will receive the technical evaluation framework, VRAM calculation logic, and architectural templates for setting up environment-gated provider paths for engines like Ollama and vLLM.

    The skill is designed to be engine-agnostic, supporting the most common local inference stacks including Ollama, llama.cpp, and vLLM, as well as GGUF and EXL2 quantization formats.

    While it covers standard transformer architectures and general VRAM sizing, it does not provide specific performance benchmarks for every new model release or specialized NPU optimizations.

    New in LLM Ops