Spark & Databricks Performance Reviewer

    1

    Paste PySpark code or an EXPLAIN plan and get ranked fixes for skew, shuffles, small files, UDFs and missed broadcasts.

    Free

    1 useSecurity scanned

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Spark & Databricks Performance Reviewer

    Example session with this skill installed

    My daily_sales job on Databricks got slow. Review jobs/daily_sales_job.py and this plan I saved with EXPLAIN FORMATTED: plans/slow_query_plan.txt.

    • Read your context and instructions
    • Compiled the spark & databricks

    (This is the scanner's real output on the bundled examples/ files, shortened.)

    
    # Spark performance review
    Files scanned: 2 | findings: 20 (high 6, medium 11, low 3)
    
    | high   | daily_sales_job.py:8   | Row-at-a-time Python UDF            | use built-ins / pandas_udf |
    | high   | daily_sales_job.py:20  | repartition(1) before write          | remove; optimizeWrite/autoCompact |
    | high   | daily_sales_job.py:21  | toPandas() on full DataFrame         | limit/aggregate first |
    | high   | slow_query_plan.txt:8  | BatchEvalPython in plan              | replace UDF (keeps Photon/codegen) |
    | high   | slow_query_plan.txt:18 | Small files: 18432 files, avg 2.3 MiB| OPTIMIZE + autoCompact / liquid clustering |
    | high   | slow_query_plan.txt:20 | Skew: max is 649x median             | AQE skewJoin, salt hot key, null keys |
    | medium | daily_sales_job.py:17  | Window.orderBy without partitionBy   | add partitionBy |
    | medium | slow_query_plan.txt:3  | SortMergeJoin                        | broadcast(stores) (~5k rows) |
    | medium | slow_query_plan.txt:1  | No AdaptiveSparkPlan                 | enable AQE |
    
    

    The agent then confirms each finding in context and proposes the top 3 changes: (1) replace clean_region with F.upper(F.trim("region")); (2) orders.join(F.broadcast(stores), "store_id"); (3) drop repartition(1) and partitionBy("order_id") (a high-cardinality key), then use CLUSTER BY (order_date) and OPTIMIZE. It ends with a before/after measurement table for you to fill in. The numbers in the sample plan are illustrative inputs, not results.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Most Spark tuning skills are a table of tips. This one reads your actual evidence.

    • Reads code and plans. A bundled read-only scanner (Python stdlib, no installs) checks .py/.sql/.scala files for 20 anti-patterns, and parses EXPLAIN FORMATTED output and pasted Spark UI metrics.
    • Uses your own numbers. It flags skew when max task shuffle is at least 5x the median. It flags small files when the average file is under 32 MiB across more than 100 files. It also catches Cartesian or nested-loop joins, sort-merge joins that could broadcast, Python UDFs in the plan, scans with no partition pruning, missing AdaptiveSparkPlan (AQE), Photon gaps, and too many shuffle exchanges.
    • Ranks by impact. Driver OOM and Cartesian joins come first, then skew, then UDFs, small files, joins and config.
    • Gives exact fixes. Code and SQL changes, null-key and salting patterns, and Delta layout advice: OPTIMIZE, auto-compaction, and liquid clustering vs Z-order vs Hive partitioning (never Z-order and liquid clustering on one table).
    • Proves it with measurement. It never promises a speed-up. It gives you a before/after table: duration, shuffle, spill, files read and cost per run, with row-count and checksum parity.

    Built by a data engineer who runs Databricks ETL in production.

    What it won't do: run jobs, change clusters, or run OPTIMIZE/VACUUM.

    Won't run jobs, change clusters, or run OPTIMIZE/VACUUM. Scanner is pattern-based and can give false positives (e.g. collect() on a tiny aggregate); the agent confirms each finding in context. Photon and liquid clustering advice applies only on Databricks.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    1 use

    Times used to date

    Free forever

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 1 month ago

    • Free with an Agensi account

    Requires

    Python 3

    Permissions

    Terminal / Shell
    Read Files

    Read: code and plan text files you provide Bash: `python scripts/review_spark.py` (Python 3 standard library, read-only, no network, no writes) No cluster, workspace or credential access

    Listed1 month ago

    What's inside

    Frequently Asked Questions

    Free ones list tips. This one parses your plan and metrics, cites file:line, ranks by impact, and makes you measure before and after.

    No. It works for Apache Spark plans too. The Photon and liquid clustering advice applies only on Databricks.

    No. Every fix comes with a measurement step.

    Possible, since it's pattern-based (e.g. `collect()` on a tiny aggregate). The agent confirms each finding in context.

    Any SKILL.md-compatible agent that can run Python 3.

    New in Data & Databases