- Home
- Skills
- Data & Databases
- Streaming and Processing at Scale
More screenshots
Works with the AI Agents you already use
Streaming and Processing at Scale
Process data with Spark, Kafka, Flink, ClickHouse, DuckDB and Polars. 10 ordered skills, starting with Apache Spark, Apache Kafka and Streaming, Apache Flink.
Free
Streaming and Processing at Scale
Example session with this skill installed
My Spark job is failing with OutOfMemory errors on the executors during a join. I have one large table and one 500MB dimension table. How do I fix this?
- Read your context and instructions
- Compiled the streaming and processing
The executor OOM during a join usually indicates a shuffle spill or an unsuccessful broadcast attempt. Given your dimension table is 500MB, it exceeds the default spark.sql.autoBroadcastJoinThreshold of 10MB.
Spark Join Optimization Plan
| Action Item | Configuration / Code Change | Reason |
|---|---|---|
| Increase Broadcast Threshold | spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "600mb") | Forces the 500MB table to be broadcast to all executors, avoiding a costly shuffle. |
| Explicit Broadcast Hint | df1.join(broadcast(df_dim), "key") | Ensures Spark attempts a BroadcastHashJoin even if metadata is missing. |
| Check for Data Skew | df1.select("key").groupBy("key").count().sort(desc("count")) | Identifies if a single key is too large for one executor's memory. |
| Increase Parallelism | spark.conf.set("spark.sql.shuffle.partitions", "auto") | Uses Adaptive Query Execution (AQE) to balance partition sizes. |
Diagnostic Verification
To confirm the fix, check the Spark UI SQL Tab:
- Locate the failed
Joinnode in the execution graph. - Verify if it shows
SortMergeJoin(bad for this case) instead ofBroadcastHashJoin. - Check Peak Execution Memory to see if the spill to disk exceeded available RAM.
Next steps
- Apply the
broadcast()hint in your PySpark code and re-run. - Enable Adaptive Query Execution with
spark.sql.adaptive.enabled=true. - Monitor the Spark UI for "Shuffle Read Size" to ensure the broadcast is working.
Connects securely to your tools. The creator never sees your data.
About this skill
The problem
Advanced data engineers often need a repeatable process, not a one-off prompt. This pack targets: Process data with Spark, Kafka, Flink, ClickHouse, DuckDB and Polars.
What's inside (10 skills, suggested learning order)
- Apache Spark — Distributed processing.
- Apache Kafka and Streaming — Real-time data.
- Apache Flink — Stateful stream processing.
- Real-Time Streaming Analytics — Real-time dashboards.
- ClickHouse — High-performance OLAP.
- DuckDB and Local Analytics — Fast analytics on files.
- Polars — High-performance DataFrames.
- IoT Data at Scale — Telemetry ingestion and analysis.
- Geospatial Data — Analyze location data.
- Web Scraping and Data Collection — Extract web data legally.
Why a skill pack instead of a one-off prompt
Each skill gives the agent a defined trigger, step-by-step procedure and output format, so results are consistent across sessions and easier to review.
How to use
- Install the pack in your SKILL.md-compatible agent.
- Ask for the task in plain language (for example: "distributed processing").
- Review the output against the skill's quality checks before using it.
Known limitations
- Outputs are drafts that need human review; they are not professional, legal or financial advice.
- Tools, APIs and prices change; check current vendor documentation before relying on version-specific details.
- Quality depends on the context you give the agent (data, requirements, constraints).
- Tested only where stated in the Compatibility note.
Also by Rodrigo Santos
Data Annotation and Labeling; Cloud Data Platforms and Data for AI; Customer and Sales Analytics
Focuses on data engineering and analytics. Does not cover hardware, firmware, or satellite imagery. Not for bypassing security controls or legal advice.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Free forever
No account required to browse
Trust & safety
Security scan
Security scan passed 4 days ago
- Free with an Agensi account