More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    HuggingFace Tokenizers: Custom Vocabularies

    1

    Train a custom tokenizer that fits your data: pick BPE, WordPiece, or Unigram in a minute, wire all four pipeline stages, and verify it before it costs you accuracy.

    $5

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    HuggingFace Tokenizers: Custom Vocabularies

    HuggingFace Tokenizers: Custom Vocabularies

    Example session with this skill installed

    We're training a small model to classify support tickets (mostly English, lots of product codes, order IDs, and the occasional pasted error log). Should we train our own tokenizer, and if so, how? Corpus is about 400 MB of historical tickets.

    • Read your context and instructions
    • Compiled the huggingface tokenizers

    Decision point 0. New model, domain-heavy text — product codes and error logs are exactly what a general tokenizer fragments. Custom tokenizer justified; keeping a stock one would bake the damage in.

    Algorithm call. BPE with byte-level pre-tokenization. WordPiece rejected: unseen product codes would fall back to [UNK] and erase the most informative tokens in the ticket. Unigram rejected: support text has clean word boundaries and doesn't repay Unigram's training cost. Byte-level guarantees every pasted log line is representable.

    Config. vocab_size 24k (narrow domain, small model — 50k would waste embedding budget), min_frequency 3 to keep one-off typos out while retaining recurring product codes, special tokens declared up front: end-of-text, pad (checked in the saved JSON that pad and end-of-text hold different IDs), plus the classification token the model family expects. Normalization: keep case (ERROR and error carry different signal in pasted logs), strip nothing else. Post-processor template matched to the classifier's expected framing.

    Checks on a held-out sample of 20k tickets. Round-trip exact modulo declared normalization. UNK rate: zero by construction. Fragmentation: the ten most common product-code families encode at 1–3 pieces; one legacy code family hit 6 pieces — the corpus sample was light on it, so historical tickets containing it were upsampled and the tokenizer retrained before training began. Speed: full corpus encodes in seconds; no slow-tokenizer surprise. Artifact pinned by hash in both the training and serving configs.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Tokenization errors are inherited by everything downstream — training, alignment, inference — and they usually come from the stages people skip: normalization chosen carelessly, no post-processor, no decoder, or a training/serving mismatch that silently shifts every token ID. This skill is the practical path through the HuggingFace tokenizers library: a 60-second algorithm decision (BPE vs WordPiece vs Unigram, with the trade-offs in one table), the training pattern with the choices that actually matter — vocabulary size, min_frequency, and training on data that looks like your inference data — and a deliberate pass through all four pipeline stages: normalize, pre-tokenize, model, post-process.

    You also get the offset-alignment workflow that makes NER and span QA work, the transformers integration rules (wrap in PreTrainedTokenizerFast, pin the exact JSON in training and serving), and a measurement discipline: speed, UNK rate, fragmentation on domain terms, and the decode-encode round-trip. A ship/no-ship rubric sets explicit thresholds for all four checks, the failure-mode table maps symptoms like UNK floods and shredded code identifiers to causes and fixes, and the worked example builds a log-analysis tokenizer that bought a classifier four accuracy points. A config template and a pre-delivery quality gate close it out.

    • Focused on the HuggingFace tokenizers library; SentencePiece-native or non-Python stacks get the principles but not the exact API calls.
    • It assumes you can supply representative training data — a tokenizer trained on the wrong corpus is the failure mode it warns about, not one it can fix without data.
    • Thresholds in the rubric are field-tested defaults for typical classification and LM work, not guarantees for every architecture.
    • It does not train or fine-tune the model itself; it ends where model training begins, with a verified tokenizer artifact.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 24 days ago

    Permissions

    No special permissions declared or detected

    Listed24 days ago
    Updated21 days ago

    What's inside

    Frequently Asked Questions

    When fine-tuning a pretrained model on text similar to its original training data. The pretrained embeddings are married to the original vocabulary; swapping tokenizers breaks that compatibility and buys nothing. The skill's first decision point exists to talk you out of unnecessary custom tokenizers.

    Most likely a train/serve mismatch: the tokenizer used at inference isn't byte-identical to the one from training, so token IDs shifted and the model sees gibberish. Pin the exact saved JSON — same hash — in both configs, and check the failure-mode table for the other usual suspects.

    30k–50k for general English, 8k–16k for narrow domains or small models, adjusted by measurement, not taste. Encode your eval set and count fragmentation: if key domain terms split into 6 or more pieces, you need more domain data or a larger vocabulary — the rubric gives the thresholds.