More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    OpenRLHF: RLHF Training at Scale

    1

    Pick the alignment method your data supports — SFT, DPO, PPO, GRPO, RLOO — audit preference pairs before training, size the run with a 1% pilot, and catch reward hacking early.

    $5

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    OpenRLHF: RLHF Training at Scale

    OpenRLHF: RLHF Training at Scale

    Example session with this skill installed

    I have a 14B model SFT'd on support conversations and 30k preference pairs built from thumbs-up/thumbs-down product logs. I can get an 8-GPU node for about a week. My manager wants PPO because "that's real RLHF." Tell me the method, the audit, and the run plan.

    • Read your context and instructions
    • Compiled the openrlhf

    Method call: DPO first — and the audit matters more than the method. Thumbs-log pairs are the riskiest input in the catalog: a thumbs-up confounds correctness with speed, tone, and length, so before any training, run the four audit checks. Expect the length-bias check to bite — if "chosen" responses run much more than ~10% longer than rejected ones, you are about to train verbosity. Check comparability (do pairs differ in answer quality, or in which agent shift produced them?), decontaminate against your eval set with an n-gram pass, and audit per slice, because support data mixes intents and one poisoned slice hides inside a clean average.

    On method: your pairs already encode the preference signal an online PPO loop would sample from, and you have no separate reward signal beyond them — that is the worked example's rejection rationale almost exactly. PPO would cost roughly 3x the compute (four models in flight) for speculative gain. Write that rejection into the run record now, with the escalation trigger: if DPO plateaus on the alignment eval and spot-checks show a failure class the pairs don't cover, PPO gets reconsidered with fresh eyes.

    Run plan: verify the SFT checkpoint is solid (never RL a shaky base). DPO at beta 0.1, one epoch over the audited pairs — drop pairs that failed the audit rather than training on all 30k. Pilot at 1% of steps to confirm memory headroom and stable KL, and extrapolate wall-clock against your week of node time. Blind human spot-checks — 20 outputs, no scores visible — at 25/50/75%. Evaluate in both directions at the end: support-quality eval for the gain, a general-capability set for the alignment tax, both reported in numbers. Ship only if the readiness scorecard hits 9/12 with no zero on data audit or eval honesty.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Most alignment runs fail in the planning, not the training: a method chosen because it sounds like the "real" recipe, preference data nobody audited, a launch sized on hardware optimism, and a reward curve that climbs while the outputs quietly get worse. This skill is the discipline around OpenRLHF runs at 7B+ scale. It starts with an inputs/outputs contract and a hard stop rule — no preference pairs and no verifiable reward means the deliverable is an SFT plan, not an improvised RL stage. A data-audit step checks pair comparability, length bias, eval decontamination, and per-slice quality before anything trains. A method table plus explicit decision points maps what you have to the method it supports, with the rejected alternative written into a run record before launch. The eight-step protocol then covers the SFT-first rule, a reward-model accuracy gate for PPO/RLOO, a 1% pilot that extrapolates wall-clock while killing the run is still cheap, scheduled blind human spot-checks, reward-and-KL dashboard discipline, and a two-direction evaluation that reports the alignment tax in numbers. It includes a worked DPO example, a copy-ready run plan template, a pre-launch checklist, a readiness scorecard with a ship threshold, edge cases (tiny pair sets, gameable verifiers, checkpoint storage blowup), and the failure-mode catalog: reward hacking, KL collapse, mode collapse, reward-model worship, and hardware optimism.

    • Guidance targets the OpenRLHF stack (Ray + vLLM generation, ZeRO-style training); other frameworks differ in flags and defaults, though the data and evaluation discipline transfers.
    • Thresholds given (length-bias tolerance, pilot size, scorecard cutoffs) are field-tested starting points, not guarantees — calibrate them to your task.
    • It does not write your training configuration for you; it governs the decisions around the run.
    • Reward-model accuracy thresholds must be set for your task before the gate is meaningful.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 24 days ago

    Permissions

    No special permissions declared or detected

    Listed24 days ago
    Updated21 days ago

    What's inside

    Frequently Asked Questions

    Look at what your data supports. Preference pairs and no separate online reward signal means DPO first: cheaper, stable, and it uses the signal you already have. PPO earns its 3x cost only when DPO has plateaued and an online signal exists that the pairs don't capture. The skill makes you write the rejection rationale down before launch.

    That's the reward-hacking signature: the policy optimized the proxy — often verbosity, flattery, or hedging — instead of the quality. Check the length distribution of new outputs, read blind spot-checks without scores visible, and compare against the base-capability eval. The fix is data and reward-model work plus KL tuning, not more steps.

    No — that's a named failure mode. The preference signal drowns in the model's confusion about the task format, and you spend your pairs teaching format instead of quality. SFT first, always; RL stages refine an instruction-following model, they don't create one.

    New in Education & Learning