GPU idling due to network bottlenecks or misconfigured drivers is a common scenario—an 8-node A100 cluster delivering only 30% utilization. The root cause is rarely the cards themselves but the environment: slow interconnect, improper NCCL parameters, or unoptimized storage. We have tuned 20+ GPU clusters over 5+ years and know how to squeeze every ampere-hour. Our approach is simple: eliminate bottlenecks across the entire chain—from drivers to job scheduler.
Why hardware is only half the story
Even top-tier GPUs won't boost performance if the rest of the system is unbalanced. The NVIDIA A100 (80GB SXM) with NVLink (600 GB/s intra-node) and H100 (80GB SXM5) with HBM3 (3.35 TB/s) are powerful, but they require matching infrastructure. Without InfiniBand and a parallel filesystem (GPFS, Lustre), you get 30–50% GPU utilization instead of 85%+. We design clusters around your workloads: for LLMs with tensor parallelism, NVLink speed is critical; for data parallelism, InfiniBand bandwidth between nodes matters most.
How to verify cluster performance after tuning
After every setup we run AllReduce tests using nccl-tests. For 8× A100, expected bandwidth is >280 GB/s at 1GB message size. If lower, we hunt for the bottleneck: NUMA affinity, driver version, switch configuration. We also launch a benchmark training run for your model (e.g., GPT-2 or BERT) and compare throughput against expectations. According to NVIDIA, proper NUMA tuning can yield up to 20% improvement.
Our GPU cluster tuning process
- Audit current infrastructure and requirements—dataset size, model types, training frequency.
- Design—select GPUs, number of nodes, interconnect type, filesystem.
- Install drivers and CUDA—production versions, enable persistence mode, optimize power limits.
- Tune NCCL—fine-tune parameters, test AllReduce bandwidth (target >280 GB/s on 8× A100).
- Integrate with scheduler—Slurm for batch training or Kubernetes + GPU Operator for containerization.
- Monitoring and optimization—DCGM, Prometheus, dashboards with key metrics.
- Documentation and team training—how to launch jobs, diagnose issues.
Example: driver and CUDA installation
# Ubuntu 22.04 apt install linux-headers-$(uname -r) nvidia-driver-535 wget https://developer.download.nvidia.com/compute/cuda/12.3.0/local_installers/cuda_12.3.0_545.23.06_linux.run sh cuda_12.3.0_545.23.06_linux.run --silent --toolkit # cuDNN tar -xvf cudnn-linux-x86_64-8.9.7.29_cuda12-archive.tar.xz cp cuda/include/cudnn*.h /usr/local/cuda/include cp cuda/lib64/libcudnn* /usr/local/cuda/lib64 ldconfig nvidia-smi; nvcc --version NCCL tuning and interconnect testing
apt install libnccl2 libnccl-dev git clone https://github.com/NVIDIA/nccl-tests cd nccl-tests && make ./build/all_reduce_perf -b 1G -e 4G -f 2 -g 8 # Expected: 1GB ~280 GB/s, 4GB ~300 GB/s (algbw) Interconnect choice: InfiniBand vs Ethernet
| Parameter | InfiniBand HDR | Ethernet 100GbE |
|---|---|---|
| Bandwidth | 200 Gbps | 100 Gbps |
| Latency | ~1 µs | ~3–5 µs |
| Scaling efficiency for LLM | 85–90% | 60–70% |
| RDMA support | Native | Requires RoCEv2 |
For multi-node training with tensor parallelism, InfiniBand is mandatory. Ethernet is acceptable only for small clusters (2–4 nodes) or inference.
Configuration comparison: single-node vs multi-node
| Parameter | Single-node (8× GPU) | Multi-node (32+ GPU) |
|---|---|---|
| Interconnect | NVLink (600 GB/s) | InfiniBand HDR (200 Gbps) |
| Storage | Local NVMe | Parallel FS (Lustre) |
| Scheduler | Slurm / Kubernetes | Slurm + gang scheduling |
| Typical task | Fine-tuning LLaMA 7B | Pre-training GPT-3 175B |
NCCL tuning details
NCCL uses Tree, Ring, and NVLS algorithms. For H100, we recommend enabling NVLS (NVLink Shared) to speed up all-reduce. The parameter NCCL_ALGO=NVLS can yield 10–15% improvement. Also important is NCCL_IB_HCA to specify InfiniBand interfaces. More details can be found in the official NCCL repository.
Orchestration: Slurm or Kubernetes?
Slurm is the HPC standard, best for long batch jobs with fixed GPU count. Kubernetes + GPU Operator suits containerized, dynamic resource allocation. We help you choose and configure gang scheduling so all GPU pods launch simultaneously.
GPU Operator installation (Helm)
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia helm install gpu-operator nvidia/gpu-operator --namespace gpu-operator --create-namespace --set driver.enabled=true --set toolkit.enabled=true Example Slurm job
#!/bin/bash #SBATCH --nodes=4 #SBATCH --ntasks-per-node=8 #SBATCH --gres=gpu:8 #SBATCH --partition=a100 #SBATCH --time=48:00:00 srun python train.py --nproc_per_node=8 --nnodes=4 Monitoring: DCGM Exporter and metrics
helm install dcgm-exporter nvidia/dcgm-exporter Key metrics: GPU utilisation (>85%), memory copy utilisation, NVLink bandwidth, power usage.
Common tuning mistakes
- Skipping NUMA affinity configuration—costs 10–20% performance.
- Using a single filesystem partition for both datasets and checkpoints—creates an IO bottleneck.
- Not running AllReduce tests between nodes—often only discovered in production.
- Wrong scheduler parameters (timeout, backfill)—GPUs sit idle.
Results and guarantees
After tuning, your cluster will deliver:
- GPU utilization ≥85% under standard loads.
- Scaling efficiency of 85–90% for multi-node training.
- Documented deployment and monitoring procedures.
We guarantee stable operation and provide support under a service agreement. We will estimate your project within 1–2 days. Contact us for a consultation and get a preliminary assessment. Order tuning and forget about GPU downtime.