GPU instances for AI are expensive: an A100 hour costs a lot, and idle time eats your budget. Cold start of a new pod takes 3–10 minutes (model loading, CUDA initialization), and standard CPU autoscaling does not account for GPU specifics. Without proper autoscaling, you pay for idle time or lose requests during peaks. We solve this: we configure scaling that actually meets SLA and reduces costs by 30–40%.
Problems We Solve
Cold start: GPU pod startup time 3–10 minutes. During this, the request queue overflows. Solutions: keepalive pods (minimum 1), pre-warming (start at 70% queue load), and buffering via request queue. Using spot instances worsens the problem — they can be interrupted anytime, so we combine spot with on-demand for critical services.
GPU utilization vs queue depth: GPU load during a long request is 100%, but new requests wait. Relying on utilization is a mistake. The correct metric is vllm_num_requests_waiting or queue_depth. We use a combination: scale-up by queue depth, scale-down by utilization.
Thrashing pods: Frequent spin-ups and downs due to sharp spikes. Configuring stabilization windows prevents this: scale-down after 10 minutes, scale-up within 30 seconds.
Why Queue Depth Is the Main Metric for LLM?
GPU utilization does not reflect queue latency. When a model processes a long request, GPU is 100% busy, but new requests are queued. Queue depth shows the real resource need. It should be the primary trigger for scale-up.
How We Configure GPU Autoscaling
Kubernetes HPA with Custom Metrics
Example HPA Configuration
# Prometheus Adapter for custom metrics apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: vllm-autoscaler namespace: ai-serving spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: vllm-llama3 minReplicas: 1 maxReplicas: 8 metrics: # Main metric: queue of waiting requests - type: Pods pods: metric: name: vllm_pending_requests target: type: AverageValue averageValue: "5" # scale when > 5 requests in queue per pod # Additional: GPU utilization (for scale-down) - type: Pods pods: metric: name: nvidia_gpu_duty_cycle target: type: AverageValue averageValue: "70" # scale-down when < 70% utilization behavior: scaleUp: stabilizationWindowSeconds: 30 # fast scale-up policies: - type: Pods value: 2 periodSeconds: 60 # +2 pods every minute scaleDown: stabilizationWindowSeconds: 600 # slow scale-down (10 minutes) policies: - type: Pods value: 1 periodSeconds: 300 # -1 pod every 5 minutes KEDA for Event-Driven Autoscaling
KEDA is more flexible than HPA: scaling by Prometheus, Kafka, RabbitMQ, SQS. Below is an example configuration.
apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: vllm-keda-scaler namespace: ai-serving spec: scaleTargetRef: name: vllm-llama3 minReplicaCount: 1 maxReplicaCount: 10 cooldownPeriod: 300 triggers: - type: prometheus metadata: serverAddress: http://prometheus.monitoring.svc.cluster.local:9090 metricName: vllm_queue_size query: sum(vllm_num_requests_waiting{namespace="ai-serving"}) threshold: "10" # 1 replica per 10 waiting requests - type: prometheus metadata: serverAddress: http://prometheus.monitoring.svc.cluster.local:9090 metricName: request_rate query: rate(http_requests_total{job="vllm"}[2m]) threshold: "20" # additional trigger by RPS Cloud-Native Autoscaling
For cloud GPU instances, we use Auto Scaling Groups with custom metrics:
import boto3 autoscaling = boto3.client('autoscaling', region_name='us-east-1') autoscaling.put_scaling_policy( AutoScalingGroupName='llm-gpu-asg', PolicyName='scale-on-queue-depth', PolicyType='TargetTrackingScaling', TargetTrackingConfiguration={ 'CustomizedMetricSpecification': { 'MetricName': 'LLMQueueDepth', 'Namespace': 'Custom/LLMMetrics', 'Statistic': 'Average', }, 'TargetValue': 5.0, 'ScaleInCooldown': 300, 'ScaleOutCooldown': 60, 'DisableScaleIn': False, } ) Publishing Custom Metrics from vLLM
We collect metrics directly from the inference server:
from prometheus_client import Gauge, start_http_server import requests import time QUEUE_SIZE = Gauge('llm_queue_depth', 'Number of pending requests') GPU_MEMORY = Gauge('llm_gpu_memory_used_gb', 'GPU memory usage in GB', ['gpu_id']) def collect_metrics(): response = requests.get("http://localhost:8000/metrics").text for line in response.split('\n'): if 'vllm:num_requests_waiting' in line and not line.startswith('#'): queue_size = float(line.split()[-1]) QUEUE_SIZE.set(queue_size) import subprocess result = subprocess.run( ['nvidia-smi', '--query-gpu=memory.used', '--format=csv,noheader,nounits'], capture_output=True, text=True ) for i, mem_mb in enumerate(result.stdout.strip().split('\n')): GPU_MEMORY.labels(gpu_id=str(i)).set(float(mem_mb) / 1024) start_http_server(9091) while True: collect_metrics() time.sleep(15) How Pre-Warming Prevents Cold Start?
We prevent cold start by forecasting load. If queue depth exceeds 70% of the maximum, we launch an additional pod in advance. Strategy code:
class PreWarmingStrategy: def __init__(self, warmup_threshold: float = 0.7, warmup_lead_time: int = 180): self.warmup_threshold = warmup_threshold self.warmup_lead_time = warmup_lead_time def should_scale_up(self, current_queue: int, max_queue: int, forecast: list) -> bool: if current_queue / max_queue >= self.warmup_threshold: return True future_queue = forecast[self.warmup_lead_time // 15] return future_queue / max_queue >= self.warmup_threshold Why GPU Autoscaling Is Harder Than CPU?
Key differences: GPU instances cannot be "split" between services, cold start is much longer (model loading ~5 GB), and load metrics are misleading. On CPU you rely on CPU utilization, on GPU — only on queue depth and request arrival rate. Also, the cost of error is higher: an extra GPU pod means extra monthly expenses.
How to Choose a Metric for Scaling?
| Metric | Description | Suitable for |
|---|---|---|
| GPU utilization | Fraction of time cores are busy, easy to collect | Scale-down, baseline monitoring |
| Queue depth (pending requests) | Number of requests in queue, requires custom exporter | Scale-up, primary metric |
| Request rate (RPS) | Request arrival rate, good for forecasting | Pre-warming, additional trigger |
| GPU memory usage | Video memory usage, low variability | Overload notifications |
The best solution is a combination: queue depth for scale-up and GPU utilization for scale-down.
What Our Work Includes
We offer turnkey implementation:
- Infrastructure audit — load analysis, bottleneck identification, instance selection.
- Scaling scheme design — metric selection, HPA/KEDA configuration, behavior optimization (stabilization windows, policies).
- Implementation — Prometheus stack deployment, vLLM/TGI integration, cloud autoscaling group setup.
- Testing — load testing with peak simulation, threshold calibration.
- Documentation and training — runbook for on-call, metric and alert description.
- Post-release support — monitoring for first weeks, policy adjustments based on actuals.
Timeline and Savings
Estimated timeline:
- Basic setup (metrics + HPA) — from 5 days.
- Extended configuration (KEDA + pre-warming) — from 10 days.
- Full cycle with load testing — from 3 weeks.
Cost is calculated individually based on stack complexity and workload. Our clients save 30–40% monthly GPU costs after implementation.
Order GPU autoscaling implementation and start saving within a week. We have been working with AI infrastructure for 5+ years, completed 20+ projects. We guarantee stable operation — we include SLA on scaling response time in the contract. Kubernetes HPA and KEDA are proven tools. Contact us for a consultation on your project — we will analyze your load and offer the optimal scheme.