token-budget-tiktoken — Tokenizer Integration
Integrate encoding-specific token counting into token-budget application code with async setup, explicit encodings and focused regression checks.
$1
Works with the AI Agents you already use
token-budget-tiktoken — Tokenizer Integration
Example session with this skill installed
Use the published adapter with cl100k_base to count hello world, verify count against encoded ids, and explain how this differs from total request usage.
- Read your context and instructions
- Compiled the token-budget-tiktoken
The published @shivam.dixit/token-budget-tiktoken 0.1.4 tarball was exercised in Node. Awaiting createTiktokenTokenizer({encoding:cl100k_base}) returned a tokenizer whose count(hello world) was 2 and encode(hello world) returned [15339,1917]. Their lengths agree. resolveEncodingForModel(gpt-4o) returned o200k_base, demonstrating why an explicit encoding and a model-derived encoding must be distinguished. Integrate the initialized tokenizer at TokenBudget construction and reuse it across calls. Keep model capacity, output reserve, per-message framing and modalities separately accounted for. This does not measure hidden prompts, change the separate MCP server's estimator, certify billing totals or establish real-world savings. Verify code, JSON and multilingual fixtures in the target application before rollout.
Connects securely to your tools. The creator never sees your data.
About this skill
Original integration workflows for @shivam.dixit/token-budget-tiktoken: identify the message-loop integration point, select an explicit tiktoken encoding, initialize the default JS tokenizer once, pass it to TokenBudget and validate representative fixtures. Includes API setup, native-path selection, migration and troubleshooting.
Requires Node.js >=18 for the tested Node path, the separate core library and tokenizer adapter. Published adapter 0.1.4 was smoke-tested from its npm tarball: cl100k_base counted hello world as 2 tokens with ids [15339,1917], and the gpt-4o mapping resolved to o200k_base. Separately, current source adapter 0.1.5 passed 30 tests. Those are distinct baselines.
This is an application library integration, not an MCP server or automatic live-context monitor. Encoding-specific text counts do not include every provider's framing, tools, hidden prompts or multimodal usage. The inspected token-budget MCP API does not accept a custom tokenizer. No billing accuracy or cost-savings guarantee is made. The optional native entry needs tiktoken and is Node-only; default JS is the portable starting point.
The runtime remains free and MIT licensed: https://github.com/shivam039/token-budget. This original MIT workflow package supplies curated guidance, not exclusive access. No runtime dependencies are bundled.
Encoding-specific text count, not exact total request billing or host context. No MCP custom-tokenizer configuration. Published 0.1.4 smoke versus source 0.1.5 suite are separate. Bundler compatibility and all dependency security were not universally certified.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 19 days ago
Requires
Permissions
Allowed Hosts
Install only in an authorized project; review commands/config and endpoints before execution. Read task-authorized files and write only authorized project changes/reports. Runtime and source packages remain free and open-source under MIT; this purchase is for original curated instructions.