More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Ray Data Pipeline Design: Scale Without Rewriting

    1

    Design and tune distributed data pipelines with Ray Data — for datasets that outgrow single-machine pandas, preprocessing that feeds trainers, and pipelines that run on a laptop and a cluster.

    $5

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Ray Data Pipeline Design: Scale Without Rewriting

    Ray Data Pipeline Design: Scale Without Rewriting

    Example session with this skill installed

    Preprocessing for a training job takes 9 hours on one machine and the dataset grows weekly. Design a Ray Data pipeline that runs on a laptop now and a cluster later, and size it.

    • Read your context and instructions
    • Compiled the ray data pipeline

    Fit check first: the dataset is 400 GB and growing, preprocessing is CPU-bound, and the trainer needs a steady feed — Ray Data fits, and single-machine pandas does not. The pipeline is staged as a lazy graph: read in streaming blocks, map the cleaning functions, filter the bad rows, and write sharded output — no stage materializes the whole dataset. The sizing example does the arithmetic: at 8 workers the map stage is the bottleneck, so the plan starts with 16 workers on the cluster and 2 on the laptop, with the same code and a different worker count. Failure modes are named up front: a straggler block from skewed keys, backpressure when the trainer stalls, and object-store pressure from oversized blocks — each with its signature and its fix. Projected runtime is 40 minutes on the cluster against the current 9 hours, with the laptop run at 2 workers taking about 5 hours for a smoke test. The pipeline was written once, with the worker count and the block size as parameters — the laptop and the cluster run the same code, and the sizing note records the numbers so the next dataset gets sized instead of guessed. When the dataset grew to 600 GB the following month, the sizing note projected the new runtime without a fresh design session — the pipeline scaled by changing two numbers, and the projection landed within ten percent of the measured run.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    A pipeline that works on a laptop and dies on a cluster was never designed, it was hoped for. This skill designs and tunes distributed data pipelines with Ray Data: the lazy-graph mental model, streaming blocks, batch inference at scale, preprocessing that feeds Ray Train or Torch and TensorFlow trainers, and the tuning checklist that turns a slow pipeline into a fast one. It starts with the fit question — Ray Data is the wrong tool for small data, and the skill says when — then covers the two recipes that answer most jobs, the failure modes and their signatures, and a worked sizing example that shows the arithmetic before the cluster runs. The promise is portability: the same pipeline runs on a laptop today and a cluster tomorrow, with the differences named instead of discovered.

    It designs pipelines; it does not run your cluster or pay for it — sizing is arithmetic, and the cluster bill is yours.
    Ray Data is the wrong tool for small data; the skill says when single-machine tools are faster and cheaper.
    Performance numbers are projections from the sizing example until measured on your data — the skill shows how to measure, not how to skip measuring.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 3 days ago

    Permissions

    No special permissions declared or detected

    Listed3 days ago

    What's inside

    Frequently Asked Questions

    When the data fits in memory and the job finishes in minutes. The overhead of distribution buys nothing there — single-machine tools are faster and cheaper.

    Yes, with the worker count and block size as parameters. The pipeline is the same; the resources change, and the sizing note records what changed and why.

    Check the failure signatures: straggler blocks mean skew, backpressure means the consumer stalls, object-store pressure means blocks are too large. Each has a distinct fix.

    New in LLM Ops