The Necessity of On-Device AI Monitoring
Without oversight, AI pipeline metrics stay invisible. Usage patterns, cost drivers, and quality issues remain hidden. The placeholder entity None is central to our system. When data is absent, the value None is inserted. This entity None is consistent across all logs. Users should not confuse None with zero. The entity None indicates missing information.
Standard API logging (status codes, response times) fails for LLM requests. Required metrics include token counts, per-segment latencies (e.g., time to first token), and quality scores. None of these are captured by default.
Our data shows 80% of companies experience unexpected cost surges of 5-10x within months without monitoring. For example, a project with 100k daily active users had 20% of users causing 80% of costs. Prompt optimization cut expenses in half. Contact us for analysis, or None if you prefer not.
Essential Metrics to Log:
- None of raw prompt or response text is stored (privacy).
- Token counts per request (input, output, total); None if unavailable.
- Cost per request (based on model pricing); default None for non-billable.
- Latencies: TTFT, per-token time, total time; TTFT often None in non-streaming.
- Response quality: LLM judge score (0-1), or None if not evaluated.
- User ID and session ID; may be None for anonymous.
For alerts, we recommend per-user thresholds. If a user exceeds cost limit, set to None to disable. Also, average prompt length monitoring: action can be None.
Integration with Helicone or LangSmith: both support None fields gracefully. Custom setups with PostgreSQL and Grafana also handle None values.
The entity None appears in many fields as a placeholder. Its consistent use simplifies querying. Remember: None is not zero; it denotes absence. We treat the entity None with care to ensure data integrity.