- Home
- Skills
- Research & Analysis
- Research Software: Correct, Tested, Reproducible Code
More screenshots
Works with the AI Agents you already use
Research Software: Correct, Tested, Reproducible Code
This skill makes your agent review and fix research code against a scientific-programming checklist, then prove the fixes with tests and an end-to-end run.
$12
Research Software: Correct, Tested, Reproducible Code
Example session with this skill installed
Review my research code and make it reproducible. scripts/train.py produces the test accuracy I report in my thesis (Table 3.2), but the number changes every time I run it and one of my tests fails. Fix what you find, add tests that pin the result, run everything at the end, and write the review to research/code-review.md. Go ahead without waiting for my confirmation. If the corrected accuracy differs from what the old script gave, tell me what the correct number is and why.
- Read your context and instructions
- Compiled the research software
The script is now reproducible and all tests pass, but the number in Table 3.2 has to change, and the corrected result shows the model has no predictive skill.
The corrected number
- Seed 0: test accuracy 0.444 (32 of 72 seeds in 6 held-out trays), 95% CI 0.264–0.653.
Across 20 seeds: 0.481 ± 0.075. Always predicting the most common class scores 0.475 on the same splits, so the model does no better than that.
- The old script gave about 0.57–0.79 depending on the run, with a mean of 0.661 across 20 seeds.
Why it dropped: the old script split individual seeds at random, so seeds from the same tray landed in both training and test. pH is the same for every seed in a tray, so it tells the model which tray a seed came from. Germination is mostly decided by the tray (11 of 24 trays germinated entirely or not at all). The forest was recognising trays it had already seen. When whole trays are held out, the accuracy falls to the baseline.
Two other bugs barely moved the number (about 0.01): the scaler was fitted on all the data before the split, and missing dry-mass values were filled with the mean of the whole table (6.09) instead of each column's mean.
Fixes
scripts/train.py: whole trays are held out for testing; imputation and scaling are fitted on the training trays only; one --seed controls all the randomness, so two runs give identical metrics.json files; the confidence interval resamples whole trays; the script reports the spread across 20 seeds and the majority-class baseline; the data path is no longer hard-coded.
- The failing test: it checked
0.1 + 0.2 == 0.3exactly. It now compares with a small tolerance.
germ/features.py: the bare except quietly turned any unreadable value into a blank. Now only real missing-value markers like n/a count as missing, and anything else raises an error.
- Setup: added packaging metadata, an editable install and
requirements-lock.txtwith the exact versions.
Tests and final run
17 tests pass. The new ones check that the data file hasn't changed, that the same seed gives the same result, the exact headline values, that no tray appears in both training and test, and a full run of the script.
- The tests catch the old mistake: when I put back the random per-seed split, 8 of them failed.
The pinned values hold only for these versions of Python, numpy and scikit-learn. I haven't tried a fresh install.
Still open
Data problem: dry mass is larger than fresh mass in 105 of 283 rows, which shouldn't be physically possible, so the dry-matter ratio feature is suspect. I didn't edit the data.
Reporting choice: with only 24 trays, report the mean ± sd across seeds next to the baseline, or use cross-validation by tray.
- pH conclusions: the effective sample size for any pH effect is 24 trays, not 288 seeds.
Full review: research/code-review.md.
Connects securely to your tools. The creator never sees your data.
About this skill
What it does
A generic code review checks style and obvious bugs. Research code fails in other ways: preprocessing fitted before the train/test split, random draws nobody seeded, tests that compare floats with ==, an except: that turns a failure into a silent NaN, a number in the paper that no script can regenerate. This skill makes your agent review and fix research code against a scientific-programming checklist, then prove the fixes with tests and an end-to-end run.
- A deterministic smell check first. The bundled
sci_check.pyparses your Python (never runs it) and reportsfile:linefindings: unseeded RNGs, exact float comparisons in tests, transformers fitted before the split or on test data, hard-coded machine paths,exceptblocks that swallow errors, packages with no tests, and results with no source script. Each finding is a lead the agent verifies by reading the code. - A review that goes past the script. The agent works through a checklist that puts science-specific failures first: group, temporal and target leakage, broadcasting, dtype and units, non-determinism, silent fallbacks, numerical stability. In the demo it found that a random row split put seeds from the same tray in train and test, a leak no static check can see.
- Fixes, then tests that pin the result. Seeds threaded through every random step, preprocessing moved inside the training fold, tolerances chosen from the metric's scale with the reason written next to them, regression tests on the headline number, seed-stability tests, and exact-by-design comparisons kept and marked rather than "fixed".
- Run it, don't trust it. The agent runs the tests and the script end to end, checks that two runs with the same seed agree, and reports the spread across seeds. In the demo it also put the old bug back to confirm the new tests catch it.
- A review you can act on.
research/code-review.md(when you ask for it): must-fix and should-fix items withfile:line, the fix and its status, what was verified with which command, and what was not checked.
The smell checker
sci_check.py needs Python 3.10+ and the standard library only. It makes no network calls and writes nothing. --fail-on high exits 1 on any HIGH finding, so it drops into CI or pre-commit. Swallowed errors (an except whose body only passes, continues or returns a default) are HIGH; a bare except: that still handles the error is a WARN. A reviewed false positive is silenced with # sci-check: ignore[CODE] on that line.
What's included
SKILL.md: the doctrine and the six-step workflow (scope, pre-check, review, fix and test, run end to end, report)references/: the review checklist, test discipline (tolerances, regression pins, seed stability, property tests), a leakage taxonomy with fixes, numerics and performance, reproducibility, CI and release gates, and the exact rules of the smell checkerscripts/sci_check.pyand a README
Part of the Aurora Lab research pipeline
Works on any Python codebase, with or without a research/ folder. Pairs well with Aurora Lab's Citable Python Packages (free), which turns scripts into an installable, citable package, and with Instructions Writer for a README a newcomer can follow; neither is needed.
The smell checker is a static heuristic for Python: it parses code and never runs it, so it cannot see a seed set in another module, semantic leakage (duplicates across splits, group or temporal leakage, target leakage, tuning on the test set), wrong units, broadcasting or dtype errors, or wrong maths, and it does not read notebooks (export them to .py first). It also flags float comparisons that are exact by design, such as a stored setting read back or a dyadic fraction; check each one and silence it with an inline # sci-check: ignore[FLOAT-EQ]. A clean run proves nothing about logic: the agent's review covers what the script cannot, but it is a careful reading, not a proof of correctness. Regression pins hold for the library versions they were recorded with; a fresh-environment install is a separate step the agent recommends but cannot guarantee in every setup. It does not scaffold packages, write up results or edit your manuscript, and it never changes your data.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 29 days ago
Requires
Permissions
File Scopes
Terminal: runs the bundled smell checker (static, reads files only), your test suite and your scripts on a small input, inside your project's environment. Reads your code, tests, packaging files and README; reads research/results/*.json only when asked to check that each result names a source script. Edits code and tests only when you ask for fixes, and writes research/code-review.md only when you ask for a written review. Never rewrites results, figures or manuscript files, never installs anything globally, never uses the network, and writes nothing outside the project except throwaway scratch files in the system temp directory (for example, two runs' outputs to compare).