More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    DeepSpeed: Distributed Training That Scales

    1

    Train large models across multiple GPUs with DeepSpeed — ZeRO stages, mixed precision, and parallelism chosen by measurement, one dimension at a time.

    $5

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    DeepSpeed: Distributed Training That Scales

    DeepSpeed: Distributed Training That Scales

    Example session with this skill installed

    We have a 13B-parameter model and 8× A100 80GB GPUs. Training OOMs at ZeRO-1 and crawls at ZeRO-3. Give me the configuration and the reasoning.

    • Read your context and instructions
    • Compiled the deepspeed

    13B on 8× A100 80GB — configuration and reasoning

    Sizing first: parameters in BF16 are about 26GB — they fit on one GPU. The problem is Adam optimizer states in FP32, roughly 100GB+. ZeRO-1 shards only the optimizer and still OOMs because gradients plus activations do not fit either. ZeRO-3 shards everything but the communication overhead is why it crawls — you are paying full parameter-sharding cost for a model that does not need it.

    Configuration: ZeRO-2 with CPU offload of optimizer states, BF16, micro-batch 4, gradient accumulation 8 (effective batch 256), gradient clipping 1.0. ds_report run first — fused kernels and async I/O verified before the run starts.

    If it still OOMs: enable gradient checkpointing before reaching for ZeRO-3 — checkpointing trades compute for activation memory, which is what sharding does not cover, and it is cheaper than ZeRO-3's communication tax.

    If throughput is poor: profile communication vs compute. If communication dominates, reduce the stage or raise the micro-batch — do not add GPUs to a communication-bound run. If it diverges: check BF16 vs FP16 first, then rescale the learning rate for the new effective batch size. Rubric score: 7/8 — the remaining point is the throughput baseline, which can only be measured on the first real run.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    DeepSpeed exists because models got bigger than GPUs — but most distributed-training failures are configuration errors, not hardware limits. This skill teaches the levers in the right order: ZeRO stages (use the lowest stage that fits; each stage up trades memory for communication), mixed precision (BF16 on Ampere-and-newer, FP32 for loss and clipping, learning-rate rescaling when the effective batch changes), and parallelism (data parallel first, pipeline for deep models, tensor for wide layers — each added one at a time with throughput measured per step).

    You get a minimal ZeRO-2 configuration with every choice justified, a decision table for the seven common situations (OOM, hang, throughput collapse, NaN loss), and a debugging section that starts with ds_report before any GPU-hours burn. A worked decision path sizes a 13B model on 8× A100 80GB from first principles. A rubric gates the launch: below 6 out of 8, the configuration is a guess with a JSON file.

    • It does not fix a bad training recipe — learning rates, data quality, and batch-size scaling still decide whether the model learns.
    • It does not help single-GPU training meaningfully; if the model fits on one GPU, the skill says so and stops.
    • It cannot abstract away the interconnect: on slow networks, no configuration recovers lost throughput.
    • Throughput numbers are measured on your hardware, not promised by the skill — the decision table tells you what to measure, not what you will get.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 3 days ago

    Permissions

    No special permissions declared or detected

    Listed3 days ago

    What's inside

    Frequently Asked Questions

    No. Each stage up trades memory for communication. ZeRO-3 is for models whose parameters genuinely do not fit — for a 13B model on A100s, ZeRO-2 with offload is faster, and the skill shows the sizing that proves it.

    BF16 on Ampere-and-newer GPUs — wider dynamic range, far fewer loss-scaling headaches. FP16 works but watches for loss-scale underflow and NaN losses mid-training. Either way, loss computation and clipping stay FP32.

    NCCL/network or mismatched world sizes, almost always. Check that every rank agrees on the configuration — batch size, precision, world size — before anything else. And run `ds_report` first next time; it flags missing operators before GPU-hours burn.

    New in LLM Ops