More screenshots
Works with the AI Agents you already use
Speculative Decoding: Faster LLM Inference
Cut LLM decode latency with speculative decoding: pick draft, heads, or lookahead from measured acceptance, size the guess count, and verify the gain at real concurrency.
$5
Speculative Decoding: Faster LLM Inference
Example session with this skill installed
We run a 34B code assistant on one 80 GB GPU, mostly single-user sessions from an IDE plugin. Prompts are short, completions are long, and users say answers stream too slowly. An 8B model from the same family is available. Should we use speculative decoding, and how should we configure it?
- Read your context and instructions
- Compiled the speculative decoding
VERDICT: Yes, with a draft model — your shape is the favourable one, subject to one measurement first.
STEP 1, BOTTLENECK: Short prompts and long completions mean prefill is a small share and decode dominates, and single-user sessions leave the GPU under-utilised per token step. Confirm it anyway: log prefill versus decode time on 50 real IDE sessions. If decode is under 60 percent of total latency, stop here and fix the larger share instead.
STEP 2, STRATEGY: Draft model. The 8B sibling is about one quarter the target size — larger than the ideal five-to-ten-times-smaller draft, so its cost will eat some gain — but it needs no training, unlike extra prediction heads, and should beat lookahead because code from a same-family draft is usually predictable. Serve both models; keep speculation behind a runtime toggle from day one.
STEP 3, ACCEPTANCE EXPECTATION: Measure alpha on held-back IDE prompts before tuning anything. Plan on 0.70 to 0.85 for code with a same-family draft; boilerplate and repeated patterns push it up, novel algorithmic code pushes it down. If measured alpha lands below 0.5, do not proceed with this draft — a smaller or better-aligned draft matters more than any setting below.
STEP 4, SIZING: Start with K = 4. At alpha 0.75 and a draft cost near 0.25 of target cost, expected tokens per iteration are about 1 + 0.75 + 0.56 + 0.42 + 0.32, plus the bonus token, roughly 4.05, at a cost of about 1 + 4 x 0.25 = 2.0 target equivalents — an expected speedup near 2.0 times before overhead. Report the measured A/B figure, not this estimate.
STEP 5, MEASUREMENT PLAN: Record baseline tokens per second at one session and at your peak concurrent sessions. Check parity: greedy outputs must match the baseline token for token on a fixed prompt set, and sampling must pass a distribution check. Ship only if the gain holds at peak concurrency, log acceptance per session, and alert if rolling alpha falls below 0.55 — that signals your code mix has drifted away from the draft.
Connects securely to your tools. The creator never sees your data.
About this skill
Speculative decoding speeds up generation by having something cheap propose tokens and the target model verify them all in one pass, with the output distribution unchanged. The speedup in a paper rarely survives contact with a real deployment, because it depends on four things papers set for you: whether decode is actually your bottleneck, how often the target accepts the guesses, how much the guessing costs, and what concurrency you serve at. This skill works through those four in order. It starts with a bottleneck check that separates prefill from decode and rules out the cases where speculation cannot help — time-to-first-token complaints, saturated large batches, and small compute-bound models. It then chooses between the three strategies from your constraints: a small draft model when a sibling exists, extra prediction heads when you can afford a short training run, and lookahead when you need a gain with no draft and no training. The guess count is sized from the acceptance math, not copied from a default, with a worked sizing showing why acceptance rate dominates and why large guess counts stop paying. A configuration template, a pre-launch checklist, and a five-row readiness rubric make the result auditable, and the production discipline is built in: parity checks for greedy and sampling modes, A/B measurement at typical and peak concurrency, acceptance-rate monitoring per workload segment, and a tested runtime kill switch. Edge cases cover mixed chat and code traffic, long prompts, structured output, quantized pairings, and multi-GPU placement, and a failure-mode table maps lost speedups and changed outputs to causes and fixes.
- Typical gain ranges are planning figures only; your measured acceptance rate, draft cost, and concurrency decide the real speedup, and at high batch sizes speculation can be a net loss.
- It cannot create a good draft where none exists — if no sibling model fits and head training is out of budget, its honest output may be a no-go with lookahead as the only remaining option.
- Parity and distribution checks are specified as procedures, but they run in your serving stack; the skill does not execute them for you.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 24 days ago
Permissions
No special permissions declared or detected