More screenshots
Works with the AI Agents you already use
HuggingFace Tokenizers: Custom Vocabularies
Train a custom tokenizer that fits your data: pick BPE, WordPiece, or Unigram in a minute, wire all four pipeline stages, and verify it before it costs you accuracy.
$5
HuggingFace Tokenizers: Custom Vocabularies
Example session with this skill installed
We're training a small model to classify support tickets (mostly English, lots of product codes, order IDs, and the occasional pasted error log). Should we train our own tokenizer, and if so, how? Corpus is about 400 MB of historical tickets.
- Read your context and instructions
- Compiled the huggingface tokenizers
Decision point 0. New model, domain-heavy text — product codes and error logs are exactly what a general tokenizer fragments. Custom tokenizer justified; keeping a stock one would bake the damage in.
Algorithm call. BPE with byte-level pre-tokenization. WordPiece rejected: unseen product codes would fall back to [UNK] and erase the most informative tokens in the ticket. Unigram rejected: support text has clean word boundaries and doesn't repay Unigram's training cost. Byte-level guarantees every pasted log line is representable.
Config. vocab_size 24k (narrow domain, small model — 50k would waste embedding budget), min_frequency 3 to keep one-off typos out while retaining recurring product codes, special tokens declared up front: end-of-text, pad (checked in the saved JSON that pad and end-of-text hold different IDs), plus the classification token the model family expects. Normalization: keep case (ERROR and error carry different signal in pasted logs), strip nothing else. Post-processor template matched to the classifier's expected framing.
Checks on a held-out sample of 20k tickets. Round-trip exact modulo declared normalization. UNK rate: zero by construction. Fragmentation: the ten most common product-code families encode at 1–3 pieces; one legacy code family hit 6 pieces — the corpus sample was light on it, so historical tickets containing it were upsampled and the tokenizer retrained before training began. Speed: full corpus encodes in seconds; no slow-tokenizer surprise. Artifact pinned by hash in both the training and serving configs.
Connects securely to your tools. The creator never sees your data.
About this skill
Tokenization errors are inherited by everything downstream — training, alignment, inference — and they usually come from the stages people skip: normalization chosen carelessly, no post-processor, no decoder, or a training/serving mismatch that silently shifts every token ID. This skill is the practical path through the HuggingFace tokenizers library: a 60-second algorithm decision (BPE vs WordPiece vs Unigram, with the trade-offs in one table), the training pattern with the choices that actually matter — vocabulary size, min_frequency, and training on data that looks like your inference data — and a deliberate pass through all four pipeline stages: normalize, pre-tokenize, model, post-process.
You also get the offset-alignment workflow that makes NER and span QA work, the transformers integration rules (wrap in PreTrainedTokenizerFast, pin the exact JSON in training and serving), and a measurement discipline: speed, UNK rate, fragmentation on domain terms, and the decode-encode round-trip. A ship/no-ship rubric sets explicit thresholds for all four checks, the failure-mode table maps symptoms like UNK floods and shredded code identifiers to causes and fixes, and the worked example builds a log-analysis tokenizer that bought a classifier four accuracy points. A config template and a pre-delivery quality gate close it out.
- Focused on the HuggingFace tokenizers library; SentencePiece-native or non-Python stacks get the principles but not the exact API calls.
- It assumes you can supply representative training data — a tokenizer trained on the wrong corpus is the failure mode it warns about, not one it can fix without data.
- Thresholds in the rubric are field-tested defaults for typical classification and LM work, not guarantees for every architecture.
- It does not train or fine-tune the model itself; it ends where model training begins, with a verified tokenizer artifact.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 24 days ago
Permissions
No special permissions declared or detected