- Home
- Skills
- Data & Databases
- Spark & Databricks Performance Reviewer
Spark & Databricks Performance Reviewer
Paste PySpark code or an EXPLAIN plan and get ranked fixes for skew, shuffles, small files, UDFs and missed broadcasts.
Free
Works with the AI Agents you already use
Spark & Databricks Performance Reviewer
Example session with this skill installed
My daily_sales job on Databricks got slow. Review jobs/daily_sales_job.py and this plan I saved with EXPLAIN FORMATTED: plans/slow_query_plan.txt.
- Read your context and instructions
- Compiled the spark & databricks
(This is the scanner's real output on the bundled examples/ files, shortened.)
# Spark performance review
Files scanned: 2 | findings: 20 (high 6, medium 11, low 3)
| high | daily_sales_job.py:8 | Row-at-a-time Python UDF | use built-ins / pandas_udf |
| high | daily_sales_job.py:20 | repartition(1) before write | remove; optimizeWrite/autoCompact |
| high | daily_sales_job.py:21 | toPandas() on full DataFrame | limit/aggregate first |
| high | slow_query_plan.txt:8 | BatchEvalPython in plan | replace UDF (keeps Photon/codegen) |
| high | slow_query_plan.txt:18 | Small files: 18432 files, avg 2.3 MiB| OPTIMIZE + autoCompact / liquid clustering |
| high | slow_query_plan.txt:20 | Skew: max is 649x median | AQE skewJoin, salt hot key, null keys |
| medium | daily_sales_job.py:17 | Window.orderBy without partitionBy | add partitionBy |
| medium | slow_query_plan.txt:3 | SortMergeJoin | broadcast(stores) (~5k rows) |
| medium | slow_query_plan.txt:1 | No AdaptiveSparkPlan | enable AQE |
The agent then confirms each finding in context and proposes the top 3 changes: (1) replace clean_region with F.upper(F.trim("region")); (2) orders.join(F.broadcast(stores), "store_id"); (3) drop repartition(1) and partitionBy("order_id") (a high-cardinality key), then use CLUSTER BY (order_date) and OPTIMIZE. It ends with a before/after measurement table for you to fill in. The numbers in the sample plan are illustrative inputs, not results.
Connects securely to your tools. The creator never sees your data.
About this skill
Most Spark tuning skills are a table of tips. This one reads your actual evidence.
- Reads code and plans. A bundled read-only scanner (Python stdlib, no installs) checks
.py/.sql/.scalafiles for 20 anti-patterns, and parsesEXPLAIN FORMATTEDoutput and pasted Spark UI metrics. - Uses your own numbers. It flags skew when max task shuffle is at least 5x the median. It flags small files when the average file is under 32 MiB across more than 100 files. It also catches Cartesian or nested-loop joins, sort-merge joins that could broadcast, Python UDFs in the plan, scans with no partition pruning, missing AdaptiveSparkPlan (AQE), Photon gaps, and too many shuffle exchanges.
- Ranks by impact. Driver OOM and Cartesian joins come first, then skew, then UDFs, small files, joins and config.
- Gives exact fixes. Code and SQL changes, null-key and salting patterns, and Delta layout advice: OPTIMIZE, auto-compaction, and liquid clustering vs Z-order vs Hive partitioning (never Z-order and liquid clustering on one table).
- Proves it with measurement. It never promises a speed-up. It gives you a before/after table: duration, shuffle, spill, files read and cost per run, with row-count and checksum parity.
Built by a data engineer who runs Databricks ETL in production.
What it won't do: run jobs, change clusters, or run OPTIMIZE/VACUUM.
Won't run jobs, change clusters, or run OPTIMIZE/VACUUM. Scanner is pattern-based and can give false positives (e.g. collect() on a tiny aggregate); the agent confirms each finding in context. Photon and liquid clustering advice applies only on Databricks.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
1 use
Times used to date
Free forever
No account required to browse
Trust & safety
Security scan
Security scan passed 1 month ago
- Free with an Agensi account
Requires
Permissions
Read: code and plan text files you provide Bash: `python scripts/review_spark.py` (Python 3 standard library, read-only, no network, no writes) No cluster, workspace or credential access