- Home
- Skills
- Data & Databases
- Data Pipelines and Orchestration
More screenshots
Works with the AI Agents you already use
Data Pipelines and Orchestration
Build ETL/ELT with dbt, Airflow, Dagster, CDC and reverse ETL. 10 ordered skills, starting with Data Engineering, ETL/ELT Pipelines, dbt (Data Build Tool).
Free
Data Pipelines and Orchestration
Example session with this skill installed
Review my Airflow DAG for performance issues and suggest a testing plan to ensure the data load is idempotent.
- Read your context and instructions
- Compiled the data pipelines
I have reviewed the DAG and identified two critical performance bottlenecks: top-level database calls during parsing and a lack of idempotency in the task logic.
Airflow DAG Review and Test Plan
| Issue | Severity | Recommendation |
|---|---|---|
Top-level Variable.get() | High | Replace with {{ var.value.key }} Jinja template to prevent scheduler lag. |
Blind INSERT logic | High | Replace with MERGE or UPSERT using the data_interval_start as a partition key. |
| Heavy imports | Med | Move import pandas inside the Python callable to speed up DAG parsing. |
Idempotency Test Plan
import pytest
from airflow.models import DagBag
def test_dag_loading():
dagbag = DagBag(dag_folder="dags/", include_examples=False)
dag = dagbag.get_dag(dag_id="my_data_pipeline")
assert dagbag.import_errors == {}
assert dag is not None
def test_idempotent_load_logic(mock_db_context):
# Simulate running the same task twice on the same data interval
task_input = {"start": "2023-01-01", "end": "2023-01-02"}
run_1 = execute_load_task(task_input)
run_2 = execute_load_task(task_input)
# Verify no duplicate rows exist in target
assert run_1.row_count == run_2.row_count
assert get_target_count(task_input) == run_1.row_count
Next steps
- Refactor the SQL operator to use a
DELETEandINSERTpattern based on the execution date. - Move all network-heavy logic into custom Airflow Hooks.
- Implement
dbttests on the target table to catch primary key violations after the Airflow run.
Connects securely to your tools. The creator never sees your data.
About this skill
The problem
Data engineers often need a repeatable process, not a one-off prompt. This pack targets: Build ETL/ELT with dbt, Airflow, Dagster, CDC and reverse ETL.
What's inside (10 skills, suggested learning order)
- Data Engineering — Design and maintain data pipelines.
- ETL/ELT Pipelines — Move and transform data.
- dbt (Data Build Tool) — Versioned analytical transformation.
- Apache Airflow — Orchestrate pipelines.
- Data Integration via APIs — Connect sources through APIs.
- Change Data Capture (CDC) — Replicate changes in real time.
- Reverse ETL — Activate warehouse data in tools.
- Python for Data Engineering — Scripts and pipelines in Python.
- Orchestration with Dagster and Prefect — Modern pipelines with built-in observability.
- Automated Data Pipeline Testing — Ensure quality on every run.
Why a skill pack instead of a one-off prompt
Each skill gives the agent a defined trigger, step-by-step procedure and output format, so results are consistent across sessions and easier to review.
How to use
- Install the pack in your SKILL.md-compatible agent.
- Ask for the task in plain language (for example: "design and maintain data pipelines").
- Review the output against the skill's quality checks before using it.
Known limitations
- Outputs are drafts that need human review; they are not professional, legal or financial advice.
- Tools, APIs and prices change; check current vendor documentation before relying on version-specific details.
- Quality depends on the context you give the agent (data, requirements, constraints).
- Tested only where stated in the Compatibility note.
Also by Rodrigo Santos
Data Annotation and Labeling; Cloud Data Platforms and Data for AI; Customer and Sales Analytics
Requires access to database logs for CDC. Does not provide hardware-specific infrastructure cost estimations or sizing.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Free forever
No account required to browse
Trust & safety
Security scan
Security scan passed 4 days ago
- Free with an Agensi account
Requires