Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Data Pipeline Failure Diagnosis Recovery Architect

    1

    Diagnose data pipeline failures and design safe, idempotent recovery plans to restore reliable data delivery.

    $9.99

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    1 useSecurity scanned
    Data Pipeline Failure Diagnosis Recovery Architect

    Data Pipeline Failure Diagnosis Recovery Architect

    Example session with this skill installed

    I have a failed Airflow task in a dbt pipeline. The error says 'Unique constraint violation' on the sales_summary table. It started yesterday at 14:00 UTC. What should I do?

    • Read your context and instructions
    • Compiled the data pipeline failure

    I have analyzed the unique constraint violation in your Airflow dbt pipeline. The failure suggests an idempotency issue where a partial write from a previous attempt or an overlap in backfill windows is causing duplicate keys in the sales_summary table.

    Pipeline Incident Diagnosis & Recovery Plan

    ComponentAssessment
    Incident Scopesales_summary table, starting 2023-10-26 14:00 UTC
    Failure BoundaryTransformation/Load (dbt write to destination)
    Risk LevelHigh (Potential duplicate data in downstream reporting)
    Confidence85% (Common symptom of non-atomic incremental loads)

    Recovery Runbook

    Evidence Preservation: Query the sales_summary table for the affected timestamp and identify the specific duplicate keys and their inserted_at metadata.
    2.

    Safety Check: Verify if the dbt model uses incremental logic and if a unique_key is defined in the configuration.
    3. Isolation: Temporarily pause the Airflow DAG to prevent repeated failures and log bloat.
    4. Execution:
    - If using Snowflake/BigQuery: Run a manual DELETE on the affected partition or records matching the duplicate keys.
    - If using dbt: Execute a targeted dbt run --select sales_summary --full-refresh if the table size allows, or a manual merge to deduplicate.
    5.

    Validation: Run SELECT count(*), count(distinct primary_key) FROM sales_summary WHERE ... to ensure the constraint is met.

    Next steps

    • Provide the dbt model configuration for sales_summary to verify the incremental strategy.
    • Check if any upstream source system started producing duplicate records at 14:00 UTC.
    • Implement a dbt-expectations test for uniqueness to catch this before the load step in the future.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Data pipeline failures cause silent corruption, downstream reporting errors, and expensive manual cleanup. This skill provides a systematic framework to diagnose root causes across the entire data lifecycle and design safe, idempotent recovery plans.

    What it does

    • Incident triage determines the scope of data loss, affected time windows, and downstream business impact.
    • Root cause diagnosis traces failures across ingestion, transformation, orchestration, and storage layers.
    • Recovery strategy design selects the safest method among retries, resumes, replays, backfills, or restores.
    • Integrity validation defines specific checks for row counts, schema drift, and business invariants before data release.
    • Prevention planning generates post-incident reports and reliability roadmaps to close monitoring gaps.

    How it works

    1. Intake triggers an assessment of the platform, architecture, error logs, and recent deployment history.
    2. Boundary identification locates the earliest point where data behavior diverged from expectations.
    3. Safety assessment evaluates idempotency, partial commit risks, and downstream side effects.
    4. Runbook generation provides ordered execution steps with stop conditions and rollback procedures.

    Frameworks & tools

    Supports SQL, Python, Spark, and dbt. Works with orchestration tools like Airflow and cloud data platforms. Applicable to both batch and streaming architectures involving message queues and object storage.

    Why this beats prompting it yourself

    Generic prompts often suggest destructive actions like clearing checkpoints or broad retries without checking for side effects. This skill enforces a safety-first methodology that preserves evidence and ensures recovery is reversible and validated.

    Use cases

    • Diagnosing a stalled streaming pipeline with growing consumer lag and checkpoint mismatch.
    • Recovering from a transformation defect that corrupted historical partitions in a data warehouse.
    • Troubleshooting authentication failures or rate limits during high-volume source extraction.
    • Designing a safe backfill for a new data contract after schema drift occurred upstream.

    Known limitations

    Does not have direct system access and requires manual log/metric input. Cannot generate platform-specific commands without version details. Redaction of secrets must be handled by the user.

    Requires manual log/metric input. Cannot generate platform-specific commands without version details. User must redact credentials. No direct system execution.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    1 use

    Times used to date

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 8 days ago

    Permissions

    No special permissions declared or detected

    Listed8 days ago
    Updated7 days ago

    What's inside

    Frequently Asked Questions

    Diagnose data pipeline failures and design safe, idempotent recovery plans to restore reliable data delivery.

    This skill uses the open SKILL.md standard. Use it through an MCP-compatible agent connected to Agensi, and check the compatibility information on this page for confirmed agents and required capabilities.

    Connect your AI agent to Agensi through MCP, then ask it to use Data Pipeline Failure Diagnosis Recovery Architect on your task. Your agent loads the latest approved version each time. Paid skills require a purchase or active subscription. Skill files are not available for download; your licence does not allow you to save, copy or export any part of a skill.

    New in Data & Databases