ASR Eval Kit: WER/CER Benchmark & Error Analysis

    1

    Benchmark speech-to-text transcripts with WER/CER metrics, error breakdowns, and diagnostic reporting.

    Free

    Security scanned

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    ASR Eval Kit: WER/CER Benchmark & Error Analysis

    Example session with this skill installed

    Evaluate the transcripts in test_results.csv. The reference is in 'ref_text' and the hypothesis is in 'model_output'. Group the results by the 'noise_level' column.

    • Read your context and instructions
    • Compiled the asr eval kit
    • Generated the data export

    I have analyzed test_results.csv containing 450 utterances. I detected systematic differences in number formatting (digits vs words) and will apply standard normalization to ensure a fair WER calculation.

    ASR Evaluation Report

    Group (Noise Level)CountWord CountWER (%)CER (%)SER (%)InsDelSub
    Clean1501,8404.2%1.8%12.0%51260
    Moderate1501,79512.8%5.4%34.6%1845167
    High1501,81228.5%14.2%72.1%54192271
    Overall4505,44715.1%7.1%39.5%77249498

    Error Analysis

    Endpointing Deletions (High Noise): 40% of errors in the 'High' group are deletions at the start of sentences. The VAD is likely cutting off speech in high ambient noise.

    • Ref: "the weather is clear" -> Hyp: "is clear"

    Numerical Normalization: The model consistently outputs digits ("25") while the reference uses words ("twenty five"). This accounts for ~2% of the total WER.
    3. Domain Terms: Frequent substitution of technical terms (e.g., "API" recognized as "a pie").

    Next steps

    1. Review per_utt.csv to inspect the 20 utterances with the highest deletion rates.
    2. Update the normalization script to handle digit-to-word expansion.
    3. Consider fine-tuning or adding a hotword list for identified domain terms.

    test_results.csv

    CSV · data export

    Generated

    Example file from a real run - the skill writes it into your workspace.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Evaluating Automatic Speech Recognition (ASR) performance requires more than a single percentage. Raw Word Error Rate (WER) often hides systemic issues like normalization mismatches, endpointing failures, or domain-specific vocabulary gaps. This skill transforms raw transcript pairs into actionable diagnostic reports.

    What it does

    • Metric Calculation computes WER, Character Error Rate (CER), and Sentence Error Rate (SER) across datasets.
    • Error Breakdown categorizes mistakes into substitutions, deletions, and insertions to identify model behavior.
    • Text Normalization handles casing, punctuation, and number formatting to ensure fair comparisons between models.
    • Grouped Analysis segments performance by speaker, noise condition, or model version to find specific weaknesses.
    • Root Cause Identification extracts the worst-performing utterances and top word confusions for targeted debugging.

    How it works

    1. Data Inspection scans your CSV/JSONL to map reference, hypothesis, and metadata columns while checking for data quality.
    2. Normalization Alignment identifies systematic differences in digits, abbreviations, and disfluencies before scoring begins.
    3. Execution runs the localized Python evaluation script to generate comprehensive metric reports and per-utterance logs.
    4. Diagnostic Review analyzes the output to isolate whether errors stem from VAD settings, morphology, or hallucination.

    Frameworks & tools

    Uses a standalone Python 3.8+ evaluation script with zero external dependencies. Supports language-specific casing for Turkish and Azerbaijani (tr/az).

    Why this beats prompting it yourself

    Generic prompts struggle with large-scale math and consistent text normalization across thousands of rows. This skill follows a strict evaluation protocol that prevents false conclusions from small sample sizes and ensures normalization parity across all compared models.

    Use cases

    • Benchmarking two different ASR providers against a golden reference set.
    • Debugging why a model fails on specific domain terms or proper nouns.
    • Measuring the impact of a new VAD or endpointing configuration on deletion rates.
    • Generating stakeholder-ready reports for transcription quality audits.

    Known limitations

    Does not perform audio transcription or speaker diarization (DER). Requires a pre-existing file containing both reference and hypothesis text.

    Requires Python 3.8+. Does not support diarization (DER) or keyword-spotting metrics. Results with under 1,000 reference words are flagged as statistically inconclusive for small deltas.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Free forever

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 1 day ago

    • Free with an Agensi account

    Requires

    Python 3.8+

    Permissions

    Terminal / Shell
    Read Files
    Write Files
    Listed1 day ago

    What's inside

    Frequently Asked Questions

    This kit goes beyond raw Word Error Rate (WER) by offering a detailed breakdown of substitutions, deletions, and insertions, alongside text normalization to ensure your metrics aren't skewed by minor formatting differences.

    The skill includes a standalone Python evaluation script and documentation for mapping your data, allowing you to run benchmarks locally or integrate them into your existing LLM-ops pipelines.

    No, this tool is designed for post-transcription analysis; you must provide a CSV or JSONL file containing both your 'ground truth' reference text and the 'hypothesis' text from your ASR model.

    Yes, it features specialized normalization logic for Turkish and Azerbaijani (`tr`/`az`) to ensure character-level accuracy for languages with unique casing rules.

    The script is built for Python 3.8+ and is designed with zero external dependencies, making it highly portable across different environments and easy to integrate into agentic workflows.

    It is ideal for comparing ASR providers (like Whisper vs. Google), auditing transcription quality for stakeholders, and identifying if errors are caused by voice activity detection (VAD) or vocabulary gaps.

    New in LLM Ops