More screenshots
Works with the AI Agents you already use
MoE Training: Train Sparse Mixture-of-Experts Models
Train sparse Mixture-of-Experts models — make the capacity, routing, and parallelism decisions up front, then run a recipe that survives expert collapse.
$5
MoE Training: Train Sparse Mixture-of-Experts Models
Example session with this skill installed
We want a 47B-capacity model for the compute of a 13B one. Train an 8-expert MoE chat model on our budget and keep the routing healthy.
- Read your context and instructions
- Compiled the moe training
Decisions before code: 8 experts per layer with top-2 routing — the capacity target is 47B-equivalent, the compute budget is 13B-equivalent, and 8 experts at top-2 is the configuration that fits both. The router gets a load-balancing loss from step one, not as a fix after collapse. Expert parallelism is set to 4 across the available GPUs, with tensor parallelism for the attention layers. Recipe: DeepSpeed MoE configuration with the capacity factor set to 1.25 — enough headroom that token dropping stays rare, not so much that the experts sit idle. The training data mixes chat, instruction, and general text so no single expert becomes the 'everything' expert. Early warning: at step 2,000 the router histogram showed two experts taking 60% of tokens. The load-balancing loss weight was raised and the learning rate on the router was lowered; by step 5,000 the distribution was even within 10% across experts. Caught by the histogram, not by the loss curve — the loss looked fine while the routing was collapsing. Failure catalog in use: token dropping spiked once when the capacity factor was too tight for a long-context batch — the fix was raising the factor for those batches, not retraining. One expert went silent for a stretch; the diagnosis was a dead router initialization, fixed by re-initializing that expert's router weights. Result: the 8-expert model trains to the target quality in the budgeted compute, with all eight experts active and the router balanced. The training record holds the DeepSpeed config, the router histograms at three checkpoints, and the two interventions with their causes — so the next MoE run starts from a recipe that has already survived its failure modes.
Connects securely to your tools. The creator never sees your data.
About this skill
A Mixture-of-Experts model keeps dozens of feed-forward experts per layer but only activates a few per token, so you get the capacity of a 47B model for the compute of a 13B one — and the catch is that routing is a learned behavior, and learned behaviors fail in specific, diagnosable ways. This skill front-loads the decisions that prevent those failures: capacity, routing, and parallelism before any code runs, a DeepSpeed and HuggingFace recipe that survives expert collapse and token dropping, a failure catalog organized by symptom, and training discipline that keeps the run honest. The worked example trains an 8-expert chat model on a budget, with the failure modes named before they happen.
It covers MoE training with DeepSpeed and HuggingFace recipes, not the design of a new routing algorithm.
Routing failures are diagnosable, not always fixable mid-run — the skill shows how to catch them early, when a fix is still cheap.
Capacity claims are compute-equivalent, not quality-equivalent; an MoE model must still be evaluated on the same benchmarks as a dense one.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 3 days ago
Permissions
No special permissions declared or detected