More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    PyTorch FSDP2: Distributed Training

    1

    Train models too large for one GPU with PyTorch FSDP2: memory arithmetic first, bottom-up fully_shard wrapping, precision and offload knobs, DCP checkpointing, and single-GPU validation.

    $5

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    PyTorch FSDP2: Distributed Training

    PyTorch FSDP2: Distributed Training

    Example session with this skill installed

    We want to pre-train a 24B-parameter transformer on 16 A100 80GB cards across two nodes. Our first attempt with plain data parallelism OOM'd in the optimizer step. Should we use FSDP2, and if so, how should the run be set up?

    • Read your context and instructions
    • Compiled the pytorch fsdp2

    MEMORY ARITHMETIC FIRST. 24B parameters with bf16 parameters and gradients plus fp32 Adam states costs roughly 12 bytes per parameter of static state: about 288 GB. Plain data parallelism replicates all of it on every card — 288 GB into an 80 GB card is exactly why the optimizer step OOM'd, and no tuning fixes replication. Sharded across 16 ranks it is ~18 GB static per rank, leaving roughly 60 GB per card for all-gathered layers and activations. Verdict: FSDP2 is the right tool; tensor parallelism is not needed at this layer size.

    SETUP, IN ORDER. torchrun across both nodes, NCCL backend, each rank sets its device from LOCAL_RANK. Build the model on the meta device, apply fully_shard per transformer block and then the root — including the embedding table, the classic forgotten module that spikes one rank's memory. Materialize with to_empty, run init, and only then construct AdamW on the sharded parameters; building the optimizer pre-shard leaves it holding stale full-parameter references. Mixed precision: bf16 parameters with fp32 gradient reduction. Gradient checkpointing on, to buy activation headroom; reshard_after_forward stays at its default until profiling says memory is spare.

    CHECKPOINTING. Distributed Checkpoint to shared storage: each rank writes its own shard, and the run can later resume on 8 or 32 cards. Pin the torch version in the run record — DCP formats are not guaranteed stable across PyTorch releases.

    VALIDATION BEFORE THE LONG RUN. A few hundred steps at reduced scale, sharded versus single-GPU, same seed and data order: loss curves tracking within tolerance proves the sharding hooks are intact. Watch the two silent killers — any direct .forward() call in custom code (it bypasses the gather/reshard hooks) and per-rank gradient clipping, which silently under-clips compared to the global-norm form.

    Expected profile: step time bounded by cross-node all-gather. If profiling shows communication dominating, raise the per-GPU micro-batch before adding nodes.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    When parameters plus gradients plus Adam state exceed one GPU, PyTorch's FSDP2 (fully_shard) shards the model across ranks instead of replicating it — each GPU holds 1/N of the weights and all-gathers what it needs per layer. This skill starts where most failed setups went wrong: the memory arithmetic, with a fill-in worksheet that sizes static state per rank before any code changes, and a decision table for when FSDP2, tensor parallelism via DeviceMesh, CPU offload, more ranks, or a single GPU is the honest answer. It then walks the setup in dependency order — torchrun launch, bottom-up fully_shard wrapping (embeddings included), meta-device construction, the optimizer built after sharding, mixed-precision and reshard policies, DTensor-aware gradient clipping, and Distributed Checkpoint save/resume that survives world-size changes. A mandatory identical-seed comparison against a single-GPU run catches the silent failure modes, and the field catalogue of what actually breaks (OOM from unsharded parts, startup hangs, communication-bound steps) comes with causes and fixes, drawn from a fully worked 13B bring-up.

    • It assumes transformer-style architectures and NCCL-backed multi-GPU setups; exotic custom layers may need their own sharding decisions the general protocol can't make for you.
    • Throughput outcomes depend on your interconnect — the skill teaches you to profile communication vs. computation, but can't promise a step time for your fabric.
    • DCP format stability across future PyTorch versions isn't guaranteed by anyone; the skill's mitigation is version pinning, not immunity.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 24 days ago

    Permissions

    No special permissions declared or detected

    Listed24 days ago
    Updated21 days ago

    What's inside

    Frequently Asked Questions

    FSDP2 shards per-parameter with DTensors instead of flat sharded tensors, and fully_shard applies to individual modules rather than wrapping the whole model. In practice: state dicts are more inspectable, and composition with tensor parallelism via DeviceMesh is much cleaner.

    Probably not. If gradient checkpointing or quantization makes it fit, FSDP2's all-gather traffic buys you nothing. The skill's worksheet and decision table exist to make exactly this call with numbers instead of vibes.

    Yes — that's a core reason to use Distributed Checkpoint. Each rank writes its shard and loading re-shards onto the new topology. Just keep the torch version pinned, since DCP formats aren't guaranteed stable across releases.