Works with the AI Agents you already use
Hybrid Local and Cloud LLM Router with Eval Gate
Route each LLM call to Ollama, llama.cpp, LM Studio, vLLM or cloud by task, privacy, quality and cost, with code and evals. ROUTE LOCAL / TEST FIRST / KEEP CLOUD
$7
Hybrid Local and Cloud LLM Router with Eval Gate
Example session with this skill installed
Our support-desk app sends three LLM calls to a cloud model: ticket triage (120k/month), customer-field extraction with PII (40k/month), and reply drafting (30k/month). We pay $2.50 per 1M input and $10 per 1M output tokens and have one RTX 4090. Which calls should go to a local Ollama model, and give me the Python router.
- Read your context and instructions
- Compiled the hybrid local
Local/Cloud Routing Plan — support-desk
Overall verdict: ROUTE LOCAL — ticket triage passed the eval gate; PII extraction must move local for privacy but is TEST FIRST until its eval runs.
Routing policy
| Call type | Vol/mo | Tokens in/out | Sensitive | Target | Verdict |
|---|---|---|---|---|---|
| triage_ticket | 120,000 | 900 / 20 | no | local, cloud fallback | ROUTE LOCAL |
| extract_customer_fields | 40,000 | 1,500 / 150 | yes | local only | TEST FIRST |
| draft_reply | 30,000 | 3,000 / 400 | no | cloud | KEEP CLOUD |
Hardware fit (estimates, RTX 4090 24 GB)
Small class (~8B) at ~4.8 bits: 8 x 4.8 / 8 = 4.8 GB weights. KV at 8,192 ctx x 4 slots ~ 4 GB. (4.8 + 4) x 1.15 ~ 10.1 GB. Fits.
Eval gate (your 50 labeled tickets)
| Call | Cloud | Local | Schema-valid | Pass |
|---|---|---|---|---|
| triage_ticket | 47/50 (94%) | 46/50 (92%) | 50/50 | yes (within 2 pts) |
| extract_customer_fields | not run | not run | - | pending |
Cost model (your prices: $2.50 in / $10 out per 1M)
- triage: 108M x 2.50 + 2.4M x 10 = $270 + $24 = $294
- extract: 60M x 2.50 + 6M x 10 = $150 + $60 = $210
- draft: 90M x 2.50 + 12M x 10 = $225 + $120 = $345
- Today: $849/mo
- Routed: draft $345 + triage escalations $294 x 8% (assumed) = $23.52 + extract $0 = $368.52
- Savings: about $480/mo (estimate; confirm escalation rate in shadow mode)
Router scaffold (excerpt)
ROUTES = {
"triage_ticket": Route("local", LOCAL_MODEL, schema=TRIAGE_SCHEMA),
"extract_customer_fields": Route("local", LOCAL_MODEL, sensitive=True),
"draft_reply": Route("cloud", CLOUD_MODEL),
}
Escalates on timeout, bad JSON after 1 retry, or confidence < 0.7. Sensitive calls never escalate and their prompts are never logged.
Rollout
- Shadow triage for 7 days, compare labels. 2. Switch triage. 3. Run the extraction eval, then cut it over.
Connects securely to your tools. The creator never sees your data.
About this skill
Most apps send every LLM call to the same cloud model, including small jobs like labeling a ticket or pulling fields out of a form. That costs money, adds latency, and can send sensitive data off the machine when it does not need to leave.
This skill works through your app's LLM calls one by one and builds a router that sends each call to a local model (Ollama, llama.cpp server, LM Studio or vLLM) or a cloud model.
- Finds every LLM call in the repo and records tokens, output shape, latency need and data sensitivity
- Sorts calls into local candidates (classify, extract, short summaries, triage) and cloud work (multi-step reasoning, long context, critical code changes)
- Checks hardware fit with the memory math shown and labeled as an estimate
- Probes your local server's OpenAI-compatible endpoints and structured output support
- Writes a routing policy table and a Python or TypeScript scaffold: one OpenAI-compatible client, config-driven rules, timeouts, fallback to cloud on failure or low confidence, and structured logs
- Runs an eval gate so a call only routes local when quality holds on your own test set
- Builds the cost and latency model from your prices and volumes, step by step
- Gives each call type, and the plan overall, a verdict of ROUTE LOCAL, TEST FIRST or KEEP CLOUD
Sensitive data is pinned local by policy and never escalated or logged in plain text.
For developers and teams running agents or LLM features who want lower API bills or on-device privacy without guessing.
Honesty note: it does not name a "best" model, it does not invent prices or benchmark scores, and nothing gets ROUTE LOCAL without eval numbers. Third-party results, such as reported cloud-cost cuts from hybrid research, are cited as reported, not promised.
Memory and savings figures are estimates built from your inputs and must be confirmed with the eval gate and shadow mode. Endpoint features differ by server and release, so the skill probes rather than assumes. It does not recommend a specific model as best, does not fine-tune, and needs web access to re-check current docs.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 1 month ago
Requires