More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Speculative Decoding: Faster LLM Inference

    1

    Cut LLM decode latency with speculative decoding: pick draft, heads, or lookahead from measured acceptance, size the guess count, and verify the gain at real concurrency.

    $5

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Speculative Decoding: Faster LLM Inference

    Speculative Decoding: Faster LLM Inference

    Example session with this skill installed

    We run a 34B code assistant on one 80 GB GPU, mostly single-user sessions from an IDE plugin. Prompts are short, completions are long, and users say answers stream too slowly. An 8B model from the same family is available. Should we use speculative decoding, and how should we configure it?

    • Read your context and instructions
    • Compiled the speculative decoding

    VERDICT: Yes, with a draft model — your shape is the favourable one, subject to one measurement first.

    STEP 1, BOTTLENECK: Short prompts and long completions mean prefill is a small share and decode dominates, and single-user sessions leave the GPU under-utilised per token step. Confirm it anyway: log prefill versus decode time on 50 real IDE sessions. If decode is under 60 percent of total latency, stop here and fix the larger share instead.

    STEP 2, STRATEGY: Draft model. The 8B sibling is about one quarter the target size — larger than the ideal five-to-ten-times-smaller draft, so its cost will eat some gain — but it needs no training, unlike extra prediction heads, and should beat lookahead because code from a same-family draft is usually predictable. Serve both models; keep speculation behind a runtime toggle from day one.

    STEP 3, ACCEPTANCE EXPECTATION: Measure alpha on held-back IDE prompts before tuning anything. Plan on 0.70 to 0.85 for code with a same-family draft; boilerplate and repeated patterns push it up, novel algorithmic code pushes it down. If measured alpha lands below 0.5, do not proceed with this draft — a smaller or better-aligned draft matters more than any setting below.

    STEP 4, SIZING: Start with K = 4. At alpha 0.75 and a draft cost near 0.25 of target cost, expected tokens per iteration are about 1 + 0.75 + 0.56 + 0.42 + 0.32, plus the bonus token, roughly 4.05, at a cost of about 1 + 4 x 0.25 = 2.0 target equivalents — an expected speedup near 2.0 times before overhead. Report the measured A/B figure, not this estimate.

    STEP 5, MEASUREMENT PLAN: Record baseline tokens per second at one session and at your peak concurrent sessions. Check parity: greedy outputs must match the baseline token for token on a fixed prompt set, and sampling must pass a distribution check. Ship only if the gain holds at peak concurrency, log acceptance per session, and alert if rolling alpha falls below 0.55 — that signals your code mix has drifted away from the draft.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Speculative decoding speeds up generation by having something cheap propose tokens and the target model verify them all in one pass, with the output distribution unchanged. The speedup in a paper rarely survives contact with a real deployment, because it depends on four things papers set for you: whether decode is actually your bottleneck, how often the target accepts the guesses, how much the guessing costs, and what concurrency you serve at. This skill works through those four in order. It starts with a bottleneck check that separates prefill from decode and rules out the cases where speculation cannot help — time-to-first-token complaints, saturated large batches, and small compute-bound models. It then chooses between the three strategies from your constraints: a small draft model when a sibling exists, extra prediction heads when you can afford a short training run, and lookahead when you need a gain with no draft and no training. The guess count is sized from the acceptance math, not copied from a default, with a worked sizing showing why acceptance rate dominates and why large guess counts stop paying. A configuration template, a pre-launch checklist, and a five-row readiness rubric make the result auditable, and the production discipline is built in: parity checks for greedy and sampling modes, A/B measurement at typical and peak concurrency, acceptance-rate monitoring per workload segment, and a tested runtime kill switch. Edge cases cover mixed chat and code traffic, long prompts, structured output, quantized pairings, and multi-GPU placement, and a failure-mode table maps lost speedups and changed outputs to causes and fixes.

    • Typical gain ranges are planning figures only; your measured acceptance rate, draft cost, and concurrency decide the real speedup, and at high batch sizes speculation can be a net loss.
    • It cannot create a good draft where none exists — if no sibling model fits and head training is out of budget, its honest output may be a no-go with lookahead as the only remaining option.
    • Parity and distribution checks are specified as procedures, but they run in your serving stack; the skill does not execute them for you.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 24 days ago

    Permissions

    No special permissions declared or detected

    Listed24 days ago
    Updated21 days ago

    What's inside

    Frequently Asked Questions

    No, when it is implemented correctly. The target model verifies every proposed token and the acceptance test preserves the target distribution, so greedy outputs should match the baseline token for token. The skill includes parity checks for both greedy and sampling modes, and treats any mismatch as a bug to disable and investigate, not as a trade-off.

    Usually nothing is broken — published ratios are typically measured at batch size one on specific hardware. At higher concurrency the GPU has less idle bandwidth, a too-large draft eats the gain, or acceptance on your real traffic is lower than on test prompts. The skill re-measures all three at your concurrency and sizes the guess count from your acceptance rate.

    Yes, through the other two strategies. Extra prediction heads need a short training run with the base model frozen, and lookahead decoding needs neither a draft nor training, in exchange for a smaller gain. The decision table in the skill maps what you have to the strategy that fits.