AI Inference Latency Monitoring: Metrics & Alerts Setup

Monitoring AI Inference Latency: Metrics & Alerts Setup

A user sends a request to an LLM and waits for the first token. If TTFT exceeds two seconds, UX drops sharply — customers leave. We, as MLOps engineers with experience in vLLM and TGI, set up latency monitoring turnkey: from metric collection to Grafana alerts. Contact us — we'll evaluate your project.

What Are P50, P95, P99 Latency and How to Interpret Them?

LLM inference latency is measured in percentiles. P50 is median latency, P95 means 95% of requests are faster than this value, P99 means 99%. For real-time services, P99 is critical; for batch, P95 is sufficient. Example: if P99 total latency > 30s, 1% of users experience unacceptable delay. Monitoring percentiles helps identify outliers and long-tail latency.

Problems We Solve

LLM inference latency consists of three components: Queuing time (time in the runtime queue), Prefill time (processing input context), and Decode time (token generation). Each requires its own metric and alert threshold. For example, long system prompts increase prefill time, requiring KV-cache caching. Comparison: vLLM with PagedAttention reduces decode latency up to 2x compared to naive implementation.

Why TTFT Monitoring Is Critical for LLM Services?

TTFT is the first sign of inference issues. If p50 TTFT > 1s, users massively leave the service. We configure alerts on p95 > 3s. Empirically: when TTFT > 5s, conversion drops by up to 40%.

How to Set Up Alerts on P99 Latency in Grafana?

We use Prometheus Alertmanager with rules based on histogram_quantile. Example alert:

---
- alert: LLMHighTTFT
  expr: histogram_quantile(0.95, rate(llm_time_to_first_token_seconds_bucket[5m])) > 3
  for: 5m
  annotations:
    summary: "TTFT p95 > 3 seconds"
- alert: LLMHighTotalLatency
  expr: histogram_quantile(0.99, rate(llm_total_latency_seconds_bucket[5m])) > 30
  for: 5m
  annotations:
    summary: "Total latency p99 > 30 seconds"
---

How We Do It: Tech Stack and Configs

The key tool is Prometheus histograms. Buckets are selected based on typical latency:

from prometheus_client import Histogram, Summary
import time

# Latency histograms
TTFT_HISTOGRAM = Histogram(
    "llm_time_to_first_token_seconds",
    "Time to first token",
    buckets=[0.1, 0.3, 0.5, 1.0, 2.0, 5.0, 10.0]
)

TOTAL_LATENCY = Histogram(
    "llm_total_latency_seconds",
    "Total request latency",
    labelnames=["model", "endpoint"],
    buckets=[0.5, 1.0, 2.0, 5.0, 10.0, 30.0, 60.0]
)

TPOT_HISTOGRAM = Histogram(
    "llm_time_per_output_token_ms",
    "Time per output token in milliseconds",
    buckets=[5, 10, 20, 50, 100, 200]
)

class LatencyTracker:
    def track_streaming_request(self, request_id: str, model: str):
        start = time.time()
        first_token_time = None

        def on_first_token():
            nonlocal first_token_time
            first_token_time = time.time()
            TTFT_HISTOGRAM.observe(first_token_time - start)

        def on_complete(total_tokens: int):
            end = time.time()
            total_latency = end - start
            TOTAL_LATENCY.labels(model=model, endpoint="/v1/chat").observe(total_latency)
            if first_token_time and total_tokens > 1:
                decode_time = end - first_token_time
                tpot_ms = (decode_time / (total_tokens - 1)) * 1000
                TPOT_HISTOGRAM.observe(tpot_ms)

        return on_first_token, on_complete

Additionally, we collect vLLM metrics: vllm:time_to_first_token_seconds, vllm:time_per_output_token_seconds, vllm:e2e_request_latency_seconds — they are already broken down by percentiles.

Metric Type Comparison

Runtime Comparison: vLLM vs TGI

What Is Included in the Work

• Documentation for all configured metrics and dashboards.
• Grafana dashboard code (JSON export).
• Deployment and alerting instructions.
• Team training: how to read dashboards and respond to alerts.
• 2 weeks of post-release support.

Estimated Timeline

From 5 to 15 working days, depending on infrastructure complexity and the number of models.

Common Mistakes

Inference Degradation Detection: Sliding Windows and Anomalies

A single latency spike is not a reason to panic. A sustained trend is a reason to act. We configure degradation detection using sliding windows:

• 7-day vs 30-day window: if the average p95 TTFT over a week grows by 30% relative to the month — automatic warning alert.
• Hourly anomalies: isolation forest on metrics per hour identifies abnormal periods (growth after a new model deployment, degradation when changing batch size).
• Correlation with GPU metrics: when TTFT increases, we check GPU utilization and memory. Increased latency + low GPU utilization = queue problem. Increased latency + high GPU memory = model doesn't fit.

Automated correlation of these metrics finds the root cause of an incident 5x faster than manual analysis.

We guarantee SLA: alert response time no more than 30 seconds. Our team's experience: 5+ years in MLOps, over 20 inference monitoring projects. Certified AWS and GCP specialists. Get a consultation — we'll evaluate your project. Order a monitoring audit — we'll prepare a plan.