Monitoring AI Inference Latency: Metrics & Alerts Setup
A user sends a request to an LLM and waits for the first token. If TTFT exceeds two seconds, UX drops sharply — customers leave. We, as MLOps engineers with experience in vLLM and TGI, set up latency monitoring turnkey: from metric collection to Grafana alerts. Contact us — we'll evaluate your project.
What Are P50, P95, P99 Latency and How to Interpret Them?
LLM inference latency is measured in percentiles. P50 is median latency, P95 means 95% of requests are faster than this value, P99 means 99%. For real-time services, P99 is critical; for batch, P95 is sufficient. Example: if P99 total latency > 30s, 1% of users experience unacceptable delay. Monitoring percentiles helps identify outliers and long-tail latency.
| Percentile | Purpose | Typical Threshold |
|---|---|---|
| P50 | Median quality | < 2s for TTFT |
| P95 | Majority of users | < 5s for TTFT |
| P99 | Edge cases | < 10s for TTFT |
Problems We Solve
LLM inference latency consists of three components: Queuing time (time in the runtime queue), Prefill time (processing input context), and Decode time (token generation). Each requires its own metric and alert threshold. For example, long system prompts increase prefill time, requiring KV-cache caching. Comparison: vLLM with PagedAttention reduces decode latency up to 2x compared to naive implementation.
Why TTFT Monitoring Is Critical for LLM Services?
TTFT is the first sign of inference issues. If p50 TTFT > 1s, users massively leave the service. We configure alerts on p95 > 3s. Empirically: when TTFT > 5s, conversion drops by up to 40%.
How to Set Up Alerts on P99 Latency in Grafana?
We use Prometheus Alertmanager with rules based on histogram_quantile. Example alert:
- alert: LLMHighTTFT expr: histogram_quantile(0.95, rate(llm_time_to_first_token_seconds_bucket[5m])) > 3 for: 5m annotations: summary: "TTFT p95 > 3 seconds" - alert: LLMHighTotalLatency expr: histogram_quantile(0.99, rate(llm_total_latency_seconds_bucket[5m])) > 30 for: 5m annotations: summary: "Total latency p99 > 30 seconds" How We Do It: Tech Stack and Configs
The key tool is Prometheus histograms. Buckets are selected based on typical latency:
from prometheus_client import Histogram, Summary import time # Latency histograms TTFT_HISTOGRAM = Histogram( "llm_time_to_first_token_seconds", "Time to first token", buckets=[0.1, 0.3, 0.5, 1.0, 2.0, 5.0, 10.0] ) TOTAL_LATENCY = Histogram( "llm_total_latency_seconds", "Total request latency", labelnames=["model", "endpoint"], buckets=[0.5, 1.0, 2.0, 5.0, 10.0, 30.0, 60.0] ) TPOT_HISTOGRAM = Histogram( "llm_time_per_output_token_ms", "Time per output token in milliseconds", buckets=[5, 10, 20, 50, 100, 200] ) class LatencyTracker: def track_streaming_request(self, request_id: str, model: str): start = time.time() first_token_time = None def on_first_token(): nonlocal first_token_time first_token_time = time.time() TTFT_HISTOGRAM.observe(first_token_time - start) def on_complete(total_tokens: int): end = time.time() total_latency = end - start TOTAL_LATENCY.labels(model=model, endpoint="/v1/chat").observe(total_latency) if first_token_time and total_tokens > 1: decode_time = end - first_token_time tpot_ms = (decode_time / (total_tokens - 1)) * 1000 TPOT_HISTOGRAM.observe(tpot_ms) return on_first_token, on_complete Additionally, we collect vLLM metrics: vllm:time_to_first_token_seconds, vllm:time_per_output_token_seconds, vllm:e2e_request_latency_seconds — they are already broken down by percentiles.
Metric Type Comparison
| Metric | What It Measures | Typical Buckets | Recommended Alert |
|---|---|---|---|
| TTFT | Time to first token | [0.1,0.3,0.5,1,2,5,10] | p95 > 3s |
| TPOT | Time per token (ms) | [5,10,20,50,100,200] | p99 > 200ms |
| Total | Total request time | [0.5,1,2,5,10,30,60] | p99 > 30s |
Runtime Comparison: vLLM vs TGI
| Parameter | vLLM | TGI |
|---|---|---|
| TTFT (p50) | ~0.3s | ~0.5s |
| Decode speed | Up to 2x faster | Stable |
| LoRA support | Yes | Yes |
| Monitoring | Built-in metrics | Prometheus exporter |
What Is Included in the Work
- Documentation for all configured metrics and dashboards.
- Grafana dashboard code (JSON export).
- Deployment and alerting instructions.
- Team training: how to read dashboards and respond to alerts.
- 2 weeks of post-release support.
Estimated Timeline
From 5 to 15 working days, depending on infrastructure complexity and the number of models.
Common Mistakes
Details on common mistakes
- Using Summary instead of Histogram — impossible to compute p99.
- Ignoring queuing time — QPS growth is masked as prefill.
- Incorrect bucket selection: too wide => loss of precision, too narrow => high cardinality.
Inference Degradation Detection: Sliding Windows and Anomalies
A single latency spike is not a reason to panic. A sustained trend is a reason to act. We configure degradation detection using sliding windows:
- 7-day vs 30-day window: if the average p95 TTFT over a week grows by 30% relative to the month — automatic warning alert.
- Hourly anomalies: isolation forest on metrics per hour identifies abnormal periods (growth after a new model deployment, degradation when changing batch size).
- Correlation with GPU metrics: when TTFT increases, we check GPU utilization and memory. Increased latency + low GPU utilization = queue problem. Increased latency + high GPU memory = model doesn't fit.
Automated correlation of these metrics finds the root cause of an incident 5x faster than manual analysis.
We guarantee SLA: alert response time no more than 30 seconds. Our team's experience: 5+ years in MLOps, over 20 inference monitoring projects. Certified AWS and GCP specialists. Get a consultation — we'll evaluate your project. Order a monitoring audit — we'll prepare a plan.