More screenshots
Works with the AI Agents you already use
BigCode Evaluation Harness: Benchmark Code Models Honestly
Benchmark code-generation models with a decision-first method: write the question first, pick benchmarks by what they measure, keep the rig honest, and read pass@k without fooling yourself.
$5
BigCode Evaluation Harness: Benchmark Code Models Honestly
Example session with this skill installed
Evaluate model M for our Python code assistant: is it good enough to ship, and is it better than our current model C? Serving temperature is 0.2. M's knowledge cutoff is June 2025. Run it honestly and give me a one-sentence verdict.
- Read your context and instructions
- Compiled the bigcode evaluation harness
EVALUATION — M ships for the paid tier; C stays for the free tier. Lead is real, cost is real, both reported.
Two questions, so two labeled experiments. Experiment 1 (ship): M on HumanEval, n=20, serving temperature 0.2, the product's chat template. Experiment 2 (comparison): M and C on the same HumanEval tasks, same settings, n=20 each.
Hygiene: M's knowledge cutoff is June 2025; HumanEval was released in 2021 — contamination assumed until shown otherwise, noted next to every number. The product's chat template recorded and fixed across both models. Sandbox: container, no network, 10-second timeout per test.
Results: Experiment 1 — M scores pass@1 68% (n=20). Experiment 2 — M 68%, C 61%: a 7-point gap at n=20, which is signal (above the 3-point noise band) but not overwhelming. Re-running the failures on HumanEval+: M drops to 63%, C to 58% — both deflate, M's lead holds at 5 points. The stricter tests didn't collapse the lead, so it was never just luck on weak tests.
Failure analysis: M's top classes are off-by-one on edge cases (8 problems) and wrong API calls (5). C's are misread spec (9) and syntax errors (6). M fails closer to correct — it understands the task and slips on details, while C misunderstands the task outright.
Cost: M costs 1.8x C per 1,000 completions. Score per dollar: M's 7-point lead costs 80% more per completion. For the product's volume, that is a real cost — flagged in the note, not hidden.
Verdict (one sentence): "M is good enough to ship and better than C on HumanEval, with a 7-point lead that survives stricter tests, at 1.8x the inference cost — ship it for the paid tier, keep C for the free tier." Generations saved, so a future re-grade needs no re-run.
Connects securely to your tools. The creator never sees your data.
About this skill
A benchmark score is an answer to a question. Most benchmark failures come from running the harness before writing down the question. This skill forces the question first, then the rig, then the reading.
Step 1 writes the question as one sentence — "good enough to ship?" needs pass@1 at the serving temperature, "better than that one?" needs a paired comparison on the same tasks and settings, "how far can it go?" needs pass@k with many samples — and if you need all three, you run three labeled experiments, not one muddled number. Step 2 chooses the benchmark by what it measures (HumanEval for quick signal, HumanEval+ to confirm, MultiPL-E for other languages, DS-1000 for data work, APPS for breadth), with a default ladder and a two-suite minimum before any decision. Step 3 is hygiene before the run: contamination (assume leakage if the model was trained after the benchmark's release, and note the cutoff next to every number), prompt format (the same model can swing 10+ points between templates — record it and keep it fixed), and execution (generated code runs in a sandbox: container, no network, timeouts).
Step 4 samples affordably: n=20 at the serving temperature for pass@1, n=200 at high temperature only for the ceiling question, and identical settings across every compared model. The rule of thumb: a 10-point gap on HumanEval at n=20 is signal; a 3-point gap is noise. Step 5 reads the results like a skeptic: group failures by class, re-run failures on the stricter variant, check the score holds in your language, and report score per dollar alongside the raw score. Step 6 produces a one-page evaluation note per run, and generations are saved so a future re-grade needs no re-run.
- A model trained after a benchmark's release is assumed contaminated; the number is useful for relative comparisons, never as an absolute capability claim.
- One benchmark, one temperature, one n is a single data point — two suites minimum before any decision.
- A 3-point gap on HumanEval at n=20 is noise; reporting it as a win is a mistake the skill will not make.
- Published scores from other teams used different templates, n, and harness versions — they are not comparable without re-running.
- The sandbox has no network; code that fetches a package fails, and that is correct behavior.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 3 days ago
Permissions
No special permissions declared or detected