- Home
- Skills
- DevOps & Cloud
- gpu cluster scheduler
gpu cluster scheduler
Design, optimize, and troubleshoot GPU cluster scheduling across Slurm, Kubernetes, and Ray.
$5
/moSubscription · cancel anytime
Secure checkout via Stripe
- Always the latest version
- Delivered live to your agent via MCP
- Cancel anytime, access ends at period end
Works with the AI Agents you already use
gpu cluster scheduler
Example session with this skill installed
I need to submit a 2-node training job on Slurm using 8 A100s per node. Ensure tasks are bound to GPUs and memory is allocated correctly to prevent starvation.
- Read your context and instructions
- Compiled the gpu cluster scheduler
I have generated the optimized Slurm batch script for your 2-node multi-GPU training job. This configuration enforces GPU binding to prevent oversubscription and sets specific memory limits per GPU to ensure stability.
Slurm Batch Configuration
#!/bin/bash
#SBATCH --job-name=multi_node_a100_train
#SBATCH --partition=gpu_a100
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-node=8
#SBATCH --cpus-per-task=12
#SBATCH --mem-per-gpu=40G
#SBATCH --gres-flags=enforce-binding
#SBATCH --output=logs/%j.out
#SBATCH --error=logs/%j.err
# Load required modules or activate environment
module load nvidia/cuda-12.1
conda activate ml-env
# Export NCCL variables for multi-node communication
export NCCL_DEBUG=INFO
export NCCL_IB_DISABLE=0
# Execute training with srun
srun python train.py --batch_size 32 --distributed
Next steps
- Verify node availability using
sinfo -p gpu_a100before submission. - Check GPU topology with
nvidia-smi topo -mto confirm NVLink health. - Monitor real-time utilization with
sstat -j <jobid> --format=AveCPU,AveRSS,MaxRSS.
Subscription · always the latest version
- Secure checkout via Stripe
- Cancel anytime
- Delivered live via MCP
About this skill
High-performance AI workloads fail when GPU resources are mismanaged or underutilized. This skill provides the engineering expertise to design, configure, and optimize GPU clusters across Slurm, Kubernetes, and Ray environments.
What it does
- Multi-scheduler orchestration handles job submission and queue management for Slurm, Kubernetes, and Ray.
- Resource partitioning configures MIG (Multi-Instance GPU) and time-slicing for multi-tenant efficiency.
- Performance optimization tunes GPU topology, NVLink/InfiniBand bindings, and data loading to maximize throughput.
- Utilization analytics monitors SM engines, HBM bandwidth, and power draw using DCGM and Prometheus.
- Advanced troubleshooting resolves OOM errors, scheduling stalls, and interconnect bottlenecks.
How it works
- Analyze cluster state by auditing node inventory, GPU topology, and existing queue policies.
- Define resource requirements using GRES flags for Slurm or resource limits for Kubernetes YAML manifests.
- Execute scheduling logic to deploy batch training jobs, distributed RL tasks, or inference services.
- Monitor and tune live workloads using
nvidia-smiqueries and DCGM metrics to identify bottlenecks.
Frameworks & tools
NVIDIA GPU Operator, DCGM, Slurm (HPC), Kubernetes (Volcano, Kueue), Ray Train, NCCL, InfiniBand, and RoCE.
Why this beats prompting it yourself
Generic AI lacks the specific syntax for complex GRES bindings and often misses critical topology considerations like NVLink domains. This skill applies battle-tested HPC patterns for preemption and fair-share that prevent cluster starvation.
Use cases
- Configuring a multi-tenant A100 cluster using MIG for isolated inference workloads.
- Optimizing multi-node LLM training jobs to use InfiniBand instead of falling back to TCP.
- Implementing Volcano batch scheduling on Kubernetes for gang-scheduled AI workloads.
Known limitations
Does not support proprietary cloud-only schedulers like AWS SageMaker or Google Vertex AI. Requires cluster-level permissions to modify configurations.
Requires root/admin access for scheduler config. Excludes proprietary cloud-native schedulers. Limited to Slurm, K8s, and Ray.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Free forever
No account required to browse
Trust & safety
Security scan
Security scan passed 22 days ago