- Home
- Skills
- Data & Databases
- Data Pipeline Guard
Works with the AI Agents you already use
Data Pipeline Guard
Finds the data pipeline bugs that finish green: lost rows, reruns that duplicate data, relaxed tests, and joins that drop records.
$12
Data Pipeline Guard
Example session with this skill installed
Review my dbt project in the current folder. I'm worried about duplicate rows in the monthly_sales model after rerunning the load.
- Read your context and instructions
- Compiled the data pipeline guard
I have completed a scan of the dbt/models/ directory and evaluated the monthly_sales.sql file against the data engineering checklist. The scan identified one high-severity issue regarding idempotency.
Pipeline review: monthly_sales
Verdict: Not safe to rerun. The current INSERT logic lacks a DELETE or MERGE predicate, which will duplicate rows if the job is triggered twice for the same period.
Findings
| # | Severity | Where | Problem | Fix |
|---|---|---|---|---|
| 1 | High | monthly_sales.sql:14 | Missing idempotency. Script uses INSERT INTO without checking for existing period keys. | Implement a DELETE statement for the target period before the INSERT or switch to a dbt incremental merge strategy. |
| 2 | Medium | schema.yml | Missing unique and not_null tests on sale_id. | Add column-level tests to the dbt YAML configuration to catch duplicates at runtime. |
| 3 | Low | monthly_sales.sql:45 | Hardcoded date literal used for filtering instead of a dynamic variable. | Use {{ var('run_date') }} to ensure backfills are consistent. |
Questions for you
- Should the
monthly_salestable maintain a history of changes (SCD Type 2), or is it strictly a snapshot of the current state? - Are there specific regions that should be excluded from the aggregate, or is the
LEFT JOINonregionsintended to include nulls?
Checks to run in your warehouse
Run this to check for existing duplicates in your target table:
python3 scripts/reconcile_sql.py --source monthly_sales --key sale_id --check duplicates
Not covered
- Performance optimization of the underlying query.
- Validation of the source
raw_transactionstable schema.
Next steps
- Add the missing
uniquetests to yourschema.ymlto prevent future silent failures. - Update the model to use an incremental strategy with a
unique_key. - Run the generated reconciliation SQL to identify any duplicates already present in production.
Connects securely to your tools. The creator never sees your data.
About this skill
Pipeline Guard is a review skill for data pipelines. It targets the failures that do not crash anything: the job reports success while rows are missing, duplicated, or silently changed.
What it does
- Scans Python ingestion code, SQL scripts, dbt models and dbt YAML
- Applies 16 checks, such as swallowed errors, page size with no paging loop, appends with no rerun protection, LEFT JOINs filtered in WHERE, incremental models with no unique_key, tests set to warn, and snapshots tracking a fixed column list
- Prints a report grouped by severity (high, medium, low), each finding with the file, line, why it matters and a fix
- Generates SQL for the checks only your warehouse can answer: row counts, duplicate keys, lost keys, dropped columns, orphan fact rows and column totals. It prints the SQL and never connects to anything
- Guides Claude through a review checklist for judgment calls a scanner cannot make: column mapping, history tracking scope, backfill safety, test strictness
How you use it Ask Claude Code: "Review the pipeline in this folder", "Is this load safe to rerun?" or "Check my dbt models before I merge". You can also run the scanner yourself, and use --fail-on high to fail a CI job.
Honest limits The scanner is static, so it cannot see your data. Some rules are heuristics and are marked (check) in the report. "No findings" means the patterns were not found, not that the pipeline is correct.
Safe by design Read only. No network calls, no database connections, no files written. Python standard library only. The code is plain and commented, so you can read it before you run it.
Includes a demo project with deliberate flaws so you can see it work in one minute.
Read-only, no live DB connection. Files >1MB skipped. Heuristics need manual verification. No symlink support.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 8 days ago
Requires
Permissions
Reads files in the folder you point it at. Does not write files, run your code, connect to databases or use the network. Does not follow symlinks.