GPU-Accelerated LLM Serving on Kubernetes

We integrate Kubernetes with GPU nodes for dozens of projects — this reduces time-to-production by 2–3 times compared to bare metal. The problem of manual LLM management: GPU idle time during failures, complex scaling, chaotic updates. Our approach provides autoscaling, rolling updates, health checks, and resource isolation.

Recently, we deployed LLaMA-3-8B for a chatbot with latency requirements of p99 < 200 ms. We used vLLM with PagedAttention on two A100s. After optimization, we achieved 45 req/s and 120 ms p99. Details below.

Why Kubernetes with GPUs Is Critical for LLMs

LLMs consume up to 320 GB of memory per model. A single pod failure should not break the service. Kubernetes ensures resource isolation and automatic recovery. In our experience, a cluster with GPU nodes pays for itself in 2–3 months due to reduced downtime, saving roughly $2,000 per node monthly. The NVIDIA GPU Operator automates drivers, and the Device Plugin manages virtualization.

Cluster Preparation: NVIDIA Device Plugin and GPU Operator

The NVIDIA Device Plugin is mandatory. Install via Helm:

helm repo add nvdp https://nvidia.github.io/k8s-device-plugin helm repo update helm upgrade -i nvdp nvdp/nvidia-device-plugin \ --namespace nvidia-device-plugin --create-namespace \ --set gfd.enabled=true \ --set devicePlugin.config.sharing.timeSlicing.resources[0].name=nvidia.com/gpu \ --set devicePlugin.config.sharing.timeSlicing.resources[0].replicas=4 

Time-slicing gives up to 4 virtual GPUs per physical GPU — saving up to 40% of costs under low load, which for a 4-GPU node amounts to roughly $2,000 per month. For production, use dedicated GPUs with MIG (Multi-Instance GPU) — isolation is higher and latency more stable.

How vLLM Compares with Competitors

vLLM is 2× faster than LMDeploy on A100 with LLaMA-3-8B under the same hardware. The reason is PagedAttention and KV-cache optimization, which reduces memory fragmentation and improves continuous batching. According to vLLM documentation, PagedAttention improves memory efficiency by 2–4×. Let's compare key metrics:

Example deployment of LLaMA-3-8B with streaming and monitoring:

apiVersion: apps/v1 kind: Deployment metadata: name: vllm-llama3-8b namespace: ai-serving spec: replicas: 2 selector: matchLabels: app: vllm-llama3-8b template: metadata: labels: app: vllm-llama3-8b annotations: prometheus.io/scrape: "true" prometheus.io/port: "8080" spec: nodeSelector: nvidia.com/gpu.product: "A100-SXM4-80GB" tolerations: - key: nvidia.com/gpu operator: Exists effect: NoSchedule containers: - name: vllm image: vllm/vllm-openai:v0.5.0 command: - python3 - -m - vllm.entrypoints.openai.api_server args: - --model=/models/llama-3-8b-instruct - --tensor-parallel-size=1 - --max-model-len=8192 - --max-num-seqs=256 - --gpu-memory-utilization=0.90 - --port=8000 ports: - containerPort: 8000 name: http resources: limits: nvidia.com/gpu: "1" memory: "32Gi" cpu: "8" requests: nvidia.com/gpu: "1" memory: "24Gi" cpu: "4" readinessProbe: httpGet: path: /health port: 8000 initialDelaySeconds: 60 periodSeconds: 10 failureThreshold: 10 livenessProbe: httpGet: path: /health port: 8000 initialDelaySeconds: 120 periodSeconds: 30 volumes: - name: model-storage persistentVolumeClaim: claimName: model-storage-pvc 

How to Scale LLMs Under Load

The standard HPA based on CPU is useless. We use custom metrics — the vLLM queue size (vllm_queue_size). Example HPA with scaling policies:

apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: vllm-hpa namespace: ai-serving spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: vllm-llama3-8b minReplicas: 1 maxReplicas: 8 metrics: - type: Pods pods: metric: name: vllm_queue_size target: type: AverageValue averageValue: "10" behavior: scaleUp: stabilizationWindowSeconds: 60 policies: - type: Pods value: 1 periodSeconds: 120 scaleDown: stabilizationWindowSeconds: 300 

This configuration saves up to 30% of GPU hours under low load.

What If the Model Doesn't Fit on a Single GPU?

For models 70B+, use tensor parallelism and node affinity to ensure GPUs are on the same node. MIG partitions the GPU at the hardware level, providing deterministic compute and memory isolation for models up to 13B. Example with 4 GPUs:

resources: limits: nvidia.com/gpu: "4" memory: "320Gi" cpu: "32" affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - topologyKey: kubernetes.io/hostname 

Comparison of GPU sharing modes:

Step-by-Step Deployment Guide

1. Install NVIDIA Device Plugin via Helm as described above.
2. Create a PVC for model storage with sufficient size (at least 50 GB for a 7B model).
3. Deploy vLLM using the provided manifest, specifying the correct model and resources.
4. Set up a Service and Ingress for API access (e.g., via Istio or Nginx).
5. Configure HPA with custom metrics by scraping them with Prometheus and the adapter.
6. Test scaling by running load tests (e.g., with Locust).

What's Included in Our Work

• Audit of current infrastructure and recommendations for GPU nodes.
• Installation and configuration of NVIDIA Device Plugin / GPU Operator.
• Deployment of vLLM with streaming and monitoring (Prometheus + metrics).
• Configuration of HPA based on custom metrics.
• Documentation and team training (2 days).
• Guaranteed stable operation — 24/7 support for the first month.
• Turnkey solution: we handle everything from cluster setup to production rollout within 2 weeks.
• Free project evaluation — contact us for a preliminary assessment.

Implementation Timeline

1–2 weeks — turnkey deployment of a single model. From 1 month — multi-model cluster with CI/CD, disaster recovery, and cost optimization.

Get a Consultation

Contact our certified engineers — we will help you deploy an LLM infrastructure that can handle production loads. Experience: 50+ projects in ML infrastructure. We offer free project evaluation: just write to us. Our service includes full setup, monitoring, and ongoing support. Evaluate your project with us — we will select the optimal configuration for your tasks.