More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    NeMo Curator: Build LLM Training Data That Holds Up

    1

    Curate LLM training corpora with NeMo Curator — quality, dedup, and safety in the right order, every filter validated on samples before it runs at scale.

    $5

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    NeMo Curator: Build LLM Training Data That Holds Up

    NeMo Curator: Build LLM Training Data That Holds Up

    Example session with this skill installed

    We are fine-tuning on 40M support documents scraped from forums, docs, and tickets. Build a NeMo Curator pipeline that cuts the junk without cutting the good stuff, and prove each filter earns its place.

    • Read your context and instructions
    • Compiled the nemo curator

    Named the risk first: a support corpus fails on boilerplate, SEO spam, and outdated guidance presented as fact, so the filters were chosen for that failure, not from a default list. Exact dedup ran first — nearly free — and cut 40M documents to 31M. Quality heuristics (length bounds, symbol ratios, repetition detectors) were validated on 5,000 stratified samples with 97% removal precision before scaling, then cut the corpus to 22M. Fuzzy dedup at Jaccard 0.8 took it to 17M; semantic dedup was skipped because the fuzzy pass left no evidence of meaningful paraphrase overlap, and the cost was not justified. PII redaction ran last on the final set, with 200 redacted documents reviewed by hand to catch over-redaction of technical identifiers — order numbers that look like SSNs. Benchmark quarantine: n-gram overlap against the eval set came back 0.3%, documented, with the overlapping eval items removed from training. Final corpus 16.4M documents with a curation log holding per-stage counts, thresholds, and sample-validation numbers. The model trained on it beats the uncut-corpus baseline by 6 points on support-bench — and the team can say why, because the log answers what it was fed. The log is what makes the next corpus better instead of just smaller.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Training data quality is the highest-leverage variable in LLM training and the hardest to inspect after the fact, so this skill is curation judgment, not just tooling: which filters to run, in what order, and how to prove each one helped instead of just shrinking the corpus. It stages the pipeline as quality heuristics, then exact dedup, then fuzzy dedup, then semantic dedup only with evidence, then PII redaction and safety classifiers last — because running classifiers on ten billion tokens of exact duplicates is burning money. Every filter is validated on stratified samples before it touches the full corpus, with removal precision and recall cost recorded in a curation log, and benchmark quarantine keeps eval text out of training data so a benchmark win is a real win.

    It teaches curation judgment and the NeMo Curator pipeline, not training itself — you still run the training job and the model.
    Filter quality depends on sample validation: a heuristic tuned on the head of the distribution can still damage the long tail if samples are not stratified.
    Semantic dedup is the costliest stage; the skill shows when the evidence justifies it and when fuzzy dedup is enough.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 3 days ago

    Permissions

    No special permissions declared or detected

    Listed3 days ago

    What's inside

    Frequently Asked Questions

    Exact dedup — it is nearly free and removes pure waste. Then quality heuristics, then fuzzy dedup, then safety classifiers last. Running classifiers on ten billion tokens of duplicates is burning money.

    Validate on a stratified sample first: inspect 50 kept and 50 removed by hand, compute removal precision and recall cost, and record both in the curation log before scaling.

    Only if you ran benchmark quarantine first. N-gram-match your eval sets against the corpus and report overlap before training — a win on contaminated data is a lie you told yourself.

    New in LLM Ops