Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Data Pipeline Guard

    1

    Finds the data pipeline bugs that finish green: lost rows, reruns that duplicate data, relaxed tests, and joins that drop records.

    $12

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned

    Data Pipeline Guard

    Example session with this skill installed

    Review my dbt project in the current folder. I'm worried about duplicate rows in the monthly_sales model after rerunning the load.

    • Read your context and instructions
    • Compiled the data pipeline guard

    I have completed a scan of the dbt/models/ directory and evaluated the monthly_sales.sql file against the data engineering checklist. The scan identified one high-severity issue regarding idempotency.

    Pipeline review: monthly_sales

    Verdict: Not safe to rerun. The current INSERT logic lacks a DELETE or MERGE predicate, which will duplicate rows if the job is triggered twice for the same period.

    Findings

    #SeverityWhereProblemFix
    1Highmonthly_sales.sql:14Missing idempotency. Script uses INSERT INTO without checking for existing period keys.Implement a DELETE statement for the target period before the INSERT or switch to a dbt incremental merge strategy.
    2Mediumschema.ymlMissing unique and not_null tests on sale_id.Add column-level tests to the dbt YAML configuration to catch duplicates at runtime.
    3Lowmonthly_sales.sql:45Hardcoded date literal used for filtering instead of a dynamic variable.Use {{ var('run_date') }} to ensure backfills are consistent.

    Questions for you

    • Should the monthly_sales table maintain a history of changes (SCD Type 2), or is it strictly a snapshot of the current state?
    • Are there specific regions that should be excluded from the aggregate, or is the LEFT JOIN on regions intended to include nulls?

    Checks to run in your warehouse

    Run this to check for existing duplicates in your target table:
    python3 scripts/reconcile_sql.py --source monthly_sales --key sale_id --check duplicates

    Not covered

    • Performance optimization of the underlying query.
    • Validation of the source raw_transactions table schema.

    Next steps

    1. Add the missing unique tests to your schema.yml to prevent future silent failures.
    2. Update the model to use an incremental strategy with a unique_key.
    3. Run the generated reconciliation SQL to identify any duplicates already present in production.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Pipeline Guard is a review skill for data pipelines. It targets the failures that do not crash anything: the job reports success while rows are missing, duplicated, or silently changed.

    What it does

    • Scans Python ingestion code, SQL scripts, dbt models and dbt YAML
    • Applies 16 checks, such as swallowed errors, page size with no paging loop, appends with no rerun protection, LEFT JOINs filtered in WHERE, incremental models with no unique_key, tests set to warn, and snapshots tracking a fixed column list
    • Prints a report grouped by severity (high, medium, low), each finding with the file, line, why it matters and a fix
    • Generates SQL for the checks only your warehouse can answer: row counts, duplicate keys, lost keys, dropped columns, orphan fact rows and column totals. It prints the SQL and never connects to anything
    • Guides Claude through a review checklist for judgment calls a scanner cannot make: column mapping, history tracking scope, backfill safety, test strictness

    How you use it Ask Claude Code: "Review the pipeline in this folder", "Is this load safe to rerun?" or "Check my dbt models before I merge". You can also run the scanner yourself, and use --fail-on high to fail a CI job.

    Honest limits The scanner is static, so it cannot see your data. Some rules are heuristics and are marked (check) in the report. "No findings" means the patterns were not found, not that the pipeline is correct.

    Safe by design Read only. No network calls, no database connections, no files written. Python standard library only. The code is plain and commented, so you can read it before you run it.

    Includes a demo project with deliberate flaws so you can see it work in one minute.

    Read-only, no live DB connection. Files >1MB skipped. Heuristics need manual verification. No symlink support.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 8 days ago

    Requires

    Python 3.9 or newer
    Claude Code

    Permissions

    Terminal / Shell
    Read Files

    Reads files in the folder you point it at. Does not write files, run your code, connect to databases or use the network. Does not follow symlinks.

    Listed8 days ago

    What's inside

    Frequently Asked Questions

    No. The scanner only reads code files in the folder you choose. For checks that need data (row counts, duplicate keys, lost keys, dropped columns), it prints SQL that you run yourself in your own warehouse. No credentials are ever needed.

    No, and it does not claim to. It checks 16 known patterns of silent failure. Some are heuristics and are marked (check) in the report so you confirm them before acting. A clean report means those patterns were not found, not that the pipeline is correct.

    Python ingestion code, plain SQL files, and dbt (models and YAML). Jinja macros are not rendered, so SQL that only exists after macro expansion is not analysed.

    The scripts make no network calls and send nothing. When Claude reads your code to do the review, that code goes to the model you are running, as with any Claude Code session. If your code is confidential, run the review in a setup where code stays on your machine.

    Yes. Run scan_pipeline.py PATH --fail-on high and it exits with code 1 when a high severity finding exists. You can skip rules with --ignore, or silence a single finding in the code with a comment that includes a reason.

    Add a comment on that line or the line above, for example: pipeline-guard: ignore PG204 final select is a published view. The finding is skipped and your reason stays in the code for the next reviewer.

    New in Data & Databases