More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    TRL Fine-Tuning: SFT, DPO and RLHF Workflows

    1

    Pick the right post-training method (SFT, DPO, GRPO, PPO), gate your data before training, and ship on held-out behavior, with recipes, heuristics, and a readiness rubric.

    $5

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    TRL Fine-Tuning: SFT, DPO and RLHF Workflows

    TRL Fine-Tuning: SFT, DPO and RLHF Workflows

    Example session with this skill installed

    We have a 7B instruct model, about 40k logged support chats, and thumbs up/down from users on the bot's answers. We want it to answer in our tone, stay in scope, and stop inventing policy details. One 24GB GPU. Plan the post-training.

    • Read your context and instructions
    • Compiled the trl fine-tuning

    Method pick, from what you have: two stages. Stage 1 is SFT — the thumbs logs tell you which answers users liked, but they are not clean demonstrations, so first build 3–5k prompt→response pairs from your best resolved chats, rewritten to the tone and scope rules you want. Stage 2 is DPO, using the thumbs data as chosen/rejected pairs. PPO is ruled out on hardware alone (four models will not fit one 24GB GPU) and GRPO is ruled out because "sounds like our tone" is not a verifiable reward.

    Data gates, applied before any training: deduplicate the chat logs by intent (support logs repeat the same 200 questions thousands of times — that teaches one trick, not support). Hand-read 30 thumbs pairs: expect to find up-votes on confident wrong answers; any pair where the chosen answer invents a policy detail gets dropped, and that failure becomes an SFT example teaching the in-scope refusal instead. Length audit: check whether up-voted answers run longer — if they do, balance before DPO or the model learns length, not quality. One format contract across every example.

    Baseline first: score the current model on 300 held-out real prompts — in-scope accuracy, invented-policy rate, tone match — plus a small general instruction set to price forgetting later. Config: SFT with LoRA, LR 1e-4, 2 epochs; then DPO from the SFT checkpoint, beta 0.1, one epoch, effective batch 64. Watch the invented-policy rate at every eval, not just win-rate.

    Ship bar, set now: invented-policy rate down at least half versus baseline, tone match up, general-set regression inside 2 points. If DPO's win-rate moves but the invention rate does not, the pairs are teaching preference without facts — the fix is more refusal SFT data, not a beta sweep.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Most post-training projects fail before the first training step: the wrong method for the data on hand, preference pairs nobody ever audited, and a ship decision made by staring at training loss. This skill runs the work in the order that prevents those failures. It opens with an inputs/outputs contract and a stop rule — no pairs and no scoring function means you have a data collection problem, not a training problem. The method picker maps what you actually have (prompt-response pairs, chosen/rejected pairs, a verifiable reward, or nothing yet) to the simplest method that fits, with the standing rule that if DPO can do it, you do not pay PPO's instability and debugging tax. Four data readiness gates come next — prompt diversity, a 30-example label spot-check, a length audit against the classic longer-equals-better reward hack, and a single format contract — and a failed gate means fixing data, not tuning hyperparameters. Each method then gets a minimal working recipe: SFT as the foundation, DPO with beta treated as the whole game, GRPO for verifiable rewards without a critic, PPO with its KL discipline, and reward modeling with a calibration check before anything downstream trusts it. The run protocol baselines the starting model on a held-out set, sets the ship bar before training, and prices catastrophic forgetting on a general instruction set before shipping. A fill-in run-plan template, a five-dimension ship-readiness rubric (ship at 8+/10, no zeros on data gates or held-out evaluation), a hardware reality check from one 24GB GPU to 70B tensor parallelism, a failure-mode table, and a complete GRPO worked example close it out.

    • It plans and judges training runs; it does not execute training or ship code for your cluster.
    • Hyperparameter starting points are heuristics for common model sizes, not guarantees — your held-out metric is the arbiter.
    • It covers post-training of an existing model, not pre-training, data labeling operations, or serving.
    • Preference-based methods cannot fix a model whose base capabilities are missing; the skill will tell you to go back to SFT data instead.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 24 days ago

    Permissions

    No special permissions declared or detected

    Listed24 days ago
    Updated21 days ago

    What's inside

    Frequently Asked Questions

    Enough for the DPO stage, not the whole run. The skill has you build a small clean SFT set first, because raw liked answers are noisy demonstrations. The logs then become chosen/rejected pairs, after the hand spot-check removes confident-but-wrong winners.

    Because PPO's extra control costs a value network, a reward model in the loop, and roughly four times the debugging surface. If preference pairs can express your goal, DPO reaches it with a fraction of the failure modes. PPO is there when you have a real scoring function and the hardware — the picker table says when.

    The ship bar is written into the run plan before training: a held-out target metric plus a maximum tolerated regression on a general instruction set. The rubric then requires 8+/10 with no zeros on data gates or held-out evaluation. Feelings after the run do not move the bar.

    New in LLM Ops