Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    LLM App Production Pack

    1

    Eight agent skills and two scripts for shipping Claude and OpenAI apps: current models and prices, streaming, tool loops, caching, cost, evals, tracing, safety.

    $29

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    LLM App Production Pack

    LLM App Production Pack

    Example session with this skill installed

    Our Claude support bot's bill went up 4x after we added the policy doc to the prompt, and nothing errors. Here's yesterday's usage export (2,000 requests) and our handler. Find it and fix it.

    system: `You are Acme's support assistant. Today is ${new Date().toISOString()}. Customer: ${userName}.\n\n${SUPPORT_POLICY}`,
    
    • Read your context and instructions
    • Compiled the llm app production

    The agent loads llm-cost-control, prices the export and checks the cache columns:

    $ node estimate-cost.mjs support-usage.jsonl
    source               model              requests  uncached in  cache read  cache write   output  hit rate    cost
    support-usage.jsonl  claude-sonnet-5-5     2,000   37,040,000           0            0  879,900      0.0%  $82.88
    total $82.88  (prices checked 2026-10-13)
    

    Every request sends about 18,500 input tokens and reads nothing from cache. The prompt-caching reference names the cause: caching is a prefix match, and the system prompt starts with a timestamp and the customer's name, so no two requests share a prefix. The fix freezes the system prompt, adds breakpoints, moves per-request values after the cached prefix, and logs the hit rate so a regression is visible:

    export async function reply(history: Anthropic.MessageParam[], userName: string, question: string) {
      const res = await client.messages.create({
        model: "claude-sonnet-5-5",
        max_tokens: 2000,
        tools: TOOLS,
        cache_control: { type: "ephemeral" },
        system: [{ type: "text", text: SUPPORT_POLICY, cache_control: { type: "ephemeral" } }],
        messages: [
          ...history,
          { role: "user", content: `Customer: ${userName}. Date: ${new Date().toISOString().slice(0, 10)}.\n\n${question}` },
        ],
      });
      const u = res.usage;
      const prompt = u.input_tokens + (u.cache_creation_input_tokens ?? 0) + (u.cache_read_input_tokens ?? 0);
      console.log(JSON.stringify({ route: "support", cache_hit: prompt ? (u.cache_read_input_tokens ?? 0) / prompt : 0 }));
      return res;
    }
    

    It explains that input_tokens on Claude excludes cached tokens, which is why the hit rate is computed from the sum, and that the 5-minute TTL is right here because support traffic arrives every few seconds. After deploying, the next day's export:

    $ node estimate-cost.mjs support-usage-after.jsonl
    source                     model              requests  uncached in  cache read  cache write   output  hit rate    cost
    support-usage-after.jsonl  claude-sonnet-5-5     2,000    1,280,000  35,044,800      715,200  879,900     94.6%  $16.65
    total $16.65  (prices checked 2026-10-13)
    

    Same traffic, 80% cheaper, and cache reads no longer count toward Claude's input-token rate limit. Before suggesting a cheaper model it runs --model claude-haiku-5-5 on the same log ($4.14 at the old token counts) and says the switch needs a pass on the support golden set with run-evals.mjs first. Finally it adds a test from the caching reference that sends the same request twice and asserts the second reads from cache.

    The handler compiles with tsc --strict against @anthropic-ai/sdk 0.133.0, and both tables are the estimator's real output on those logs.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Ask an agent to build an LLM feature today and you get last year's API: temperature and budget_tokens on Claude models that now reject them with a 400, forced tool_choice that current Claude models no longer accept, Chat Completions code where OpenAI's Responses API is the default, prices and model ids that no longer exist, and a cost estimate made with the wrong tokenizer. Then production finds the rest: a timestamp in the system prompt that silently disables prompt caching, an agent loop with no cap, tool errors that crash the loop, parallel tool results split across messages, Anthropic and OpenAI usage fields summed the same way when one includes cached tokens and the other does not, 429s from a spend cap retried forever, and prompts with customer data sitting in the trace backend.

    This pack gives the agent current rules, checked on 2026-10-13 against Anthropic's and OpenAI's docs, the SDK changelogs and the packages themselves:

    • llm-app-production: model and SDK version tables, routing, the rules agents get wrong, a pre-flight checklist.
    • llm-calls: request shapes for both providers, streaming with abort, stop reasons, structured outputs with zod and pydantic, SDK retry behaviour, rate-limit headers, the 429s you must not retry.
    • llm-tool-use: tool schemas and strict mode, parallel calls in one message, errors as results, approval gates, turn and spend caps, OpenAI reasoning items, tested loops for both providers.
    • llm-cost-control: how each provider bills cache reads and writes, Claude and OpenAI prompt caching (including OpenAI's new explicit breakpoints), batch APIs, token counting, budgets, routing.
    • llm-evals: golden sets, deterministic graders first, binary judge rubrics and calibration, CI gates on baselines.
    • llm-observability: OpenTelemetry GenAI attributes mapped field by field for both providers, tool spans, safe prompt logging.
    • llm-safety: prompt injection described as patterns with layered defences, least-privilege tools, exfiltration channels, PII minimisation.
    • llm-rag: chunking, embeddings, hybrid search and reranking, citations, recall@k and faithfulness evals.

    Two zero-dependency Node scripts: estimate-cost.mjs prices API logs, batch results or a prompt file across Claude and OpenAI models, including cache writes, 1-hour TTLs and long-context tiers; run-evals.mjs grades a JSONL golden set offline or live (only when a key is already set), with LLM-as-judge and CI gates.

    TypeScript snippets typechecked with tsc --strict against @anthropic-ai/sdk 0.133 and openai 7.32; Python checked with pyright against anthropic 1.13 and openai 3.29; both agent loops run against a mock server.

    Scripts need Node.js 20+. No live API calls were made while building the pack; the agent loops were tested against a local mock server. Prices were copied from the official pricing pages on 2026-10-13 and will drift; unknown models are reported as unpriced, never guessed. OpenTelemetry GenAI conventions are still in Development status.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 1 month ago

    Requires

    Python

    Permissions

    Environment Variables
    Terminal / Shell
    Read Files
    Network Access

    File Scopes

    src/**
    evals/**
    *.jsonl

    estimate-cost.mjs only reads the usage files you pass and its price table. run-evals.mjs reads your golden set and saved outputs; it calls the Anthropic or OpenAI API only when you configure live generation or a judge and ANTHROPIC_API_KEY or OPENAI_API_KEY is already set, and sends the key only to that provider (or your *_BASE_URL).

    Listed1 month ago

    What's inside

    Frequently Asked Questions

    Model ids, prices, cache minimums and SDK majors changed this year, and agents quote old ones. The pack has current tables for the Claude and OpenAI APIs (checked 2026-10-13), the patterns that cut cost such as prompt caching and batch, and two scripts that price your real usage and gate prompt changes on evals.

    @anthropic-ai/sdk 0.133 and anthropic 1.13 for Python; openai 7.32 for Node and 3.29 for Python. Version boundaries such as Node 22 for openai 7 are noted.

    One entry skill with seven guides (API calls, tool use, cost control, evals, observability, safety, RAG) and two zero-dependency Node scripts: estimate-cost.mjs with prices.json, and run-evals.mjs.

    Only if you ask for live generation or a judge and an API key is already set. Without a key it grades saved outputs offline and reports live cases as skipped.

    No. Every guide covers both the Claude API and the OpenAI Responses and Chat Completions APIs.

    New in LLM Ops