Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Data Pipeline Architect — Build Reliable Data Movement

    1

    Design evidence-based batch, streaming and CDC pipelines with contracts, safe retries, quality gates, recovery and observability.

    $9.99

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Data Pipeline Architect — Build Reliable Data Movement

    Data Pipeline Architect — Build Reliable Data Movement

    Example session with this skill installed

    Illustrative scenario: Design a manual architecture proposal for daily order CSV ingestion into an analytics table. Each row has order_id, updated_at and amount. Files may be resent; updates can arrive in later files. We need repeatable loads and visibility into invalid rows. No volume, platform, schema rules or SLA exists yet. Do not implement or claim exactly-once delivery.

    • Read your context and instructions
    • Compiled the data pipeline architect

    Data pipeline proposal — illustrative, platform-neutral

    Requirements to confirm: authoritative source, order_id uniqueness scope, currency/amount semantics, timestamp timezone and tie-breaking, deletes, daily delivery expectation, volume, retention and recovery objectives. No SLA or platform selected without these facts.

    Flow: authorized source CSV → immutable raw landing with file checksum and ingestion metadata → parsed staging → contract/quality checks → accepted records plus quarantine → deterministic merge into the analytical target → reconciliation and load ledger. Storage and retention need approval.

    Idempotency: use file identity to detect repeat deliveries and stable business keys plus explicit version ordering to handle row updates. A resent file must not create duplicate orders. Equal timestamps with conflicting content require a documented policy or quarantine, not arbitrary overwrite.

    Quality gates: check required key, parseable timestamp, valid amount under agreed semantics and duplicate/conflicting versions. Missing amount is not automatically zero. Record rejected counts and reasons without exposing unnecessary personal data.

    Recovery: retain replayable raw input where policy permits; checkpoint only after durable accepted processing; retry deterministic operations; support scoped backfill with the same rules. Define stop conditions and rollback/forward-fix procedures before production.

    Observability: file arrival, source/data freshness, accepted/quarantined records, merge effects and authoritative reconciliation controls. A green job alone does not prove correct data.

    Validation scenarios: identical resend, later update, malformed row, equal-version conflict, partial failure before/after target commit and replay. End-to-end semantics must be tested under these failures.

    Execution notes: no files processed, infrastructure deployed or delivery guarantee demonstrated.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Data Pipeline Architect designs and reviews data movement between operational sources, APIs, files, streams, storage and analytical or application consumers. It starts with data products, source authority and contracts, then defines freshness, correctness, ordering, recovery, privacy and cost requirements.

    The PIPE-88 workflow covers ingestion, transformation, orchestration, idempotency, schema evolution, late data, backfills, checkpoints, reconciliation and lineage. It treats retries and replay as normal operating concerns and measures data correctness as well as infrastructure health. End-to-end delivery guarantees require proof under failure, not a casual exactly-once label.

    Use supplied schemas, sanitized examples, requirements and logs. Tool-dependent inspection or implementation occurs only with authorization; absent live access it produces architecture, contracts, runbooks and validation steps. It does not invent connector support, metrics, SLAs or production success.

    Correctness depends on source/sink semantics, versions, connectors, transaction behavior, schemas, workload and operations. Delivery guarantees are end-to-end properties that require failure testing. A design is not proof of production performance, recovery or correctness.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 1 month ago

    Permissions

    Read Files
    Write Files

    File Scopes

    Authorized
    task
    inputs
    and
    agreed
    output
    directory
    only

    Use authorized schemas, logs and sanitized files. Optional read/write files support design inputs and deliverables. Do not expose credentials, connection strings or confidential rows. Database/cloud access, code execution, deployment and production changes require explicit authorization and suitable tools.

    Listed1 month ago

    What's inside

    Frequently Asked Questions

    No. It defines delivery semantics and verification needs; end-to-end guarantees depend on the actual system and failure testing.

    Yes. It can produce contracts, architecture, retry/backfill plans and validation scenarios from supplied evidence.

    No. It adds quality gates, reconciliation, freshness and semantic checks beyond infrastructure job status.

    New in Data & Databases