dataforge

    3

    A manifest-driven toolkit for building, linting, and running Databricks and DuckDB data pipelines.

    Free

    2 uses5.0 (1 review)

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    dataforge

    Example session with this skill installed

    dataforge build a new metric for daily active users based on the logins table in the silver catalog, then open a PR.

    • Read your context and instructions
    • Compiled the dataforge

    I have processed the request for the daily_active_users metric. I scanned the silver.logins schema from the manifest, generated the transformation SQL, and validated the pipeline configuration.

    Metric Pipeline: Daily Active Users

    1. Transformation SQL (src/gold/m_daily_active_users.sql)

    -- Metric: Daily Active Users
    -- Description: Count of unique user_ids per day from silver.logins
    SELECT
        CAST(login_timestamp AS DATE) AS event_date,
        COUNT(DISTINCT user_id) AS dau_count
    FROM
        silver.logins
    GROUP BY
        1
    ORDER BY
        1 DESC;
    

    2. Job Configuration (conf/jobs/dau_pipeline.yaml)

    name: dau_pipeline
    tasks:
      - task_key: calculate_dau
        sql_task:
          query_path: src/gold/m_daily_active_users.sql
        warehouse_id: ${dataforge.warehouse_id}
    schedule:
      quartz_cron_expression: "0 0 2 * * ?"
      timezone_id: UTC
    

    3. Verification Results

    • SQL Syntax: PASSED (DuckDB dialect check)
    • Schema Alignment: PASSED (References silver.logins.user_id)
    • Standards Check: PASSED (Naming convention m_* enforced)

    4. Pull Request Created

    • Branch: feature/metric-dau
    • PR URL: https://github.com/org/repo/pull/42
    • Status: Open - Awaiting CI/CD

    Next steps

    1. Review the generated SQL in src/gold/m_daily_active_users.sql.
    2. Run dataforge run job dau_pipeline to test the execution in your dev environment.
    3. Merge the pull request to deploy the schedule to production.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Data engineering teams often struggle with inconsistent pipeline standards, fragmented warehouse access, and the manual toil of turning requirements into production code. Dataforge provides a manifest-driven toolkit to automate the lifecycle of data pipelines from discovery to deployment. It enforces a unified interface for interacting with your warehouse while ensuring all code adheres to project-specific standards.

    What it does

    • Warehouse Discovery scans your codebase and warehouse to generate a central manifest of truths.
    • Unified Querying executes SQL, explores catalogs, and monitors jobs across Databricks and DuckDB through one interface.
    • Automated Pipeline Building converts GitHub issues or natural language requests into full metric pipelines and pull requests.
    • Standards Enforcement lints pipeline code, SQL schemas, and job configurations against manifest definitions.
    • Platform Monitoring triggers and tracks job execution status directly within the agent environment.

    How it works

    Four cooperating agent skills that build data pipelines with no knowledge of your data baked in. Everything business-specific (zones, catalogs, domains, grains, naming) lives in one generated manifest; the skills read it and never contain it.

    1. forge-init inspects your warehouse and codebase and writes the manifest from an eight-line seed.
    2. forge-dbx is one interface for queries, jobs, and catalogs on Databricks or DuckDB.
    3. forge-lint enforces naming, SQL formatting, join strategy, and safety gates before code is written.
    4. forge-builder takes a GitHub issue to a reviewed pull request in seven phases, with a human checkpoint at each.

    What it runs

    local CLIs only (databricks or duckdb, gh, git, python for a bundled verification script). It makes no network calls of its own and asks for confirmation at every point that changes anything.

    Frameworks & tools

    This skill utilizes the databricks CLI, duckdb, gh for GitHub integration, git, and python. It is designed for data platforms running on Databricks or local DuckDB instances. Try it with no cloud account: the bundled examples/support-desk is a local DuckDB project. Run python seed.py, then say "initialize dataforge".

    Why this beats prompting it yourself

    Generic prompts often hallucinate table schemas or ignore local project conventions. Dataforge uses a resolved manifest as a single source of truth, ensuring the agent always knows your specific catalog structure and pipeline standards.

    Use cases

    • Generate a new Databricks metric pipeline from a GitHub feature request.
    • Audit existing SQL scripts against project-wide schema and naming standards.
    • Explore DuckDB catalogs and run local data transformations without leaving the IDE.
    • Initialize a new data project by scaffolding a manifest from an existing warehouse.

    Known limitations

    Requires local installation of Databricks CLI or DuckDB. All project-specific data must be defined in the manifest to ensure accuracy.

    MIT licensed

    Source, issues, and releases https://github.com/tewarishishir/dataforge

    Requires local installation of Databricks CLI, DuckDB, and GitHub CLI. Operates only on projects with a valid dataforge manifest. Beta status may involve breaking changes in manifest schema.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    5.0
    1 review
    5
    1
    4
    0
    3
    0
    2
    0
    1
    0

    This skill has been used 2 times.

    Trust & safety

    Security scan

    Security scan passed 1 month ago

    • Free with an Agensi account

    Requires

    Python
    macOS

    Permissions

    Terminal / Shell
    Network Access
    Environment Variables
    Listed1 month ago

    What's inside

    Frequently Asked Questions

    Dataforge acts as a bridge between your agent and your warehouse, using a manifest file to ensure the AI understands your specific schemas and coding standards rather than relying on generic halluncinated SQL.

    The skill requires the Databricks CLI, DuckDB, the GitHub CLI (gh), and Python to be installed in the agent's environment to execute queries and manage workflows.

    Yes, Dataforge provides a unified interface that allows you to explore catalogs and run transformations on both Databricks and DuckDB seamlessly within the same project.

    The 'Build' feature allows the agent to read a GitHub issue and automatically generate the necessary Python or SQL pipeline code, which it then validates against your project's manifest before creating a pull request.

    You will receive the complete Dataforge toolkit, including the manifest-resolution engine, warehouse discovery tools, and the suite of validation scripts for linting and monitoring.

    New in Data & Databases