More screenshots
Works with the AI Agents you already use
Fine-Tune a Small Model, Then Prove It
Teach a small model your yes/no and risk-score decisions from your own past calls — then make it beat the untuned model on data it never saw, or it doesn't ship.
$15
Fine-Tune a Small Model, Then Prove It
Example session with this skill installed
Fine-tune our small decision model on this labeled dataset, then evaluate it against the untuned baseline and tell me whether we should serve it.
- Read your context and instructions
- Compiled the fine-tune a small
Evaluation complete — held-out set, 200 labeled questions.
Baseline (untuned): accuracy 0.71, ECE 0.14, Brier 0.21
Fine-tuned (LoRA): accuracy 0.83, ECE 0.06, Brier 0.13
Accuracy gain: +0.12. Paired bootstrap 95% confidence interval on the gain: +0.07 to +0.17 — the gain is real, not noise.
Per-question deltas: 34 questions fixed, 10 regressed. The regressions are listed with their questions and labels, so you can see exactly what got worse before you decide.
Calibration: expected calibration error fell from 0.14 to 0.06 and Brier score from 0.21 to 0.13, so the tuned model's probabilities are more trustworthy, not just more often right.
Gate decision: SERVE. Accuracy, calibration, and the bootstrap interval all pass the eval-gate checklist, with regressions reviewed and acceptable. If any gate had failed, the decision would have been DO NOT SERVE, with the failing metric named.
Connects securely to your tools. The creator never sees your data.
About this skill
A decision model is not a chatbot. It takes a situation plus typed questions — allow or block, which team owns this ticket, how risky is this on a scale of 1 to 5 — and returns a probability for each answer. Fine-tuning one on your labeled examples is the easy half, and it's the half most people stop at. This skill's real subject is the proof. A tuned model is a claim, and the evaluation is where the claim gets tested.
Inside the package: the skill file covering the dataset format, the split and leakage checks, the training recipe, the metrics, and the gate; an evaluation-gate checklist; and an example dataset file with deliberately tricky guardrail cases. The dataset is one JSON object per line: the situation, the questions (yes/no, pick-one, or a numbered scale), and your answers. Training uses LoRA — a method that trains a small add-on instead of the whole model — starting from a known-good recipe for the model family. A single 24 GB graphics card handles a roughly 9B model; 50 to 200 labeled examples per question is a reasonable start, drawn from real past decisions wherever possible, because invented examples teach the model your imagination instead of your operation.
A run goes like this: hold out about a fifth of the examples as a test set with a fixed split, and check it for near-duplicates of training examples first — a test item that almost repeats a training item measures memory, not learning. Train, then run the tuned model and the untuned baseline on the same test set and read all of it: accuracy overall and per question; calibration — when the model says 80%, is it right about 80% of the time; a score that punishes confident wrong answers; the list of individual questions that got fixed and the list that got worse, read by name; and a statistical resampling check that the improvement is real and not luck on a small test set. The gate is hard and set before training: accuracy clears your floor, the improvement is statistically real, calibration did not get worse, the probability score improved or held, and you've read and accepted every regression. Fail any one and the answer is more labels for the weak spots and a retrain — not deployment.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 6 days ago