Imagine this: your LLM system suddenly starts generating dangerous responses—users see personal data in output, and your compliance officer demands an audit. You open the logs: empty. Not a single request saved. Token costs are rising, and you have no idea which model consumed what. Without structured logging, you lose control over your system. We've seen this dozens of times: companies waste hours debugging because they don't know exactly what went into the prompt. With 5+ years of MLOps experience and over 30 AI logging projects, we set up turnkey logging so you have full audit trails for every request and response. This isn't just logging—it's a system that lets you debug errors, calculate the real cost per request, comply with regulations (GDPR, CCPA, 152-FZ), and optimize prompts. Without it, you risk money and reputation.
Why Logging Is Critical for LLMs
LLMs aren't just API calls. Large data volumes, PII in prompts, high cardinality values (different user IDs, models, parameters). Without logs, you cannot:
- Debug errors: no context when something breaks.
- Track costs: each token costs money, but you don't know who spent what.
- Comply with regulations: GDPR and 152-FZ require audit trails for data processing.
We implemented a logging system for a fintech client that handles 10,000 requests per minute. After deployment, the client reduced LLM costs by 30%—we identified non-optimal prompts and lowered token count. This is a typical case: without logs, optimization is guesswork.
How PII Filtering Prevents Data Leaks
Before writing, all messages pass through a filter that masks card numbers, emails, and phone numbers. Example:
import re class PIIFilter: PATTERNS = [ (r'\b\d{4}[\s-]?\d{4}[\s-]?\d{4}[\s-]?\d{4}\b', '[CARD_NUMBER]'), (r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b', '[EMAIL]'), ] def filter(self, text: str) -> str: for pattern, replacement in self.PATTERNS: text = re.sub(pattern, replacement, text) return text Important: the filter must be exhaustive, otherwise client data gets exposed. We add custom patterns for your domain.
How to Set Up Logging: Step-by-Step
-
Choose a library. We use
structlogfor Python—it outputs clean JSON with context and integrates easily with OpenTelemetry. It includes timestamps, logger names, and levels by default—everything for centralized collection. - PII filtering. Described above.
- Choose storage. For hot logs, we use ClickHouse—it's 3–5x faster than Elasticsearch on aggregations. For cold archive, S3 Glacier with lifecycle policies. For example, warm logs (7–30 days) stored in S3 Standard, then automatically migrated to Glacier after 30 days.
- Set up metrics. We log: model, prompt_tokens, completion_tokens, latency_ms, cost_usd, error_type. For compliance, the full request body (after PII filtering).
Example structlog configuration
import structlog structlog.configure( processors=[ structlog.stdlib.filter_by_level, structlog.stdlib.add_logger_name, structlog.stdlib.add_log_level, structlog.processors.TimeStamper(fmt="iso"), structlog.processors.JSONRenderer() ], context_class=dict, logger_factory=structlog.stdlib.LoggerFactory(), cache_logger_on_first_use=True, ) Log Storage and Archival
| Tier | Retention | Storage | Purpose |
|---|---|---|---|
| Hot | <7 days | ClickHouse | Search & dashboards |
| Warm | 7–30 days | S3 Standard | Audit |
| Cold | 30–365 days | S3 Glacier | Compliance |
Expiration is handled via lifecycle rules. Storage cost is minimal: in practice, for 10,000 requests per minute, the cost is under 3% of the LLM budget. We recommend server-side AES256 encryption for all tiers.
Metrics to Monitor
| Metric | Description | Importance |
|---|---|---|
| latency_p99 | 99th percentile latency: the metric users complain about | Critical |
| cost_per_user | Cost per user: helps detect consumption anomalies | High |
| error_rate | Error rate: if it exceeds 1%, investigate | High |
| prompt_tokens | Distribution of prompt lengths: long prompts are expensive | Medium |
| cache_hit_rate | Cache hit percentage: low means caching is ineffective | Medium |
For each metric, we set up alerts in Grafana. If latency_p99 exceeds 2 seconds, you get a notification in Telegram or Slack.
What Our Turnkey Setup Includes
- Audit of current architecture: identify bottlenecks and PII leaks.
- Integration of structlog and OpenTelemetry into your Python or Node.js service.
- Deployment of ClickHouse for hot logs and S3 for archive.
- Configuration of retention, encryption (server-side AES256), and lifecycle policies.
- Grafana dashboards: latency p99, cost per user, errors by model.
- Documentation and team training.
We guarantee that after implementation, every request and response will be logged with full context. Contact us for an audit of your current logging system—our engineers will analyze your infrastructure and propose the optimal configuration. Get a demo of the architecture in two days.