AIOps with Dynatrace Davis AI: setup, calibration, integration

Dynatrace Davis AI setup: why thresholds don't catch incidents

Static thresholds are almost always set to the peak value: CPU utilization percentage, a fixed response time ceiling, an error counter from the previous incident. As long as the service operates in one mode, this works.

As soon as nightly batch processing, seasonal traffic growth, or cache warm-up after a release appears, the threshold triggers on normal behavior, and alert noise grows.

The flip side of the same configuration is missed degradations. Response time can slowly creep up twofold over an hour without ever crossing the absolute threshold: it is set with headroom for peak.

The on-call engineer sees a green dashboard while users are already writing to support. And having grown accustomed to a stream of identical notifications, the engineer stops reviewing them—and a real trigger drowns in false ones.

The reason is that a threshold knows nothing about the service's normal profile.

Dynamic baselines in Dynatrace Davis AI are built differently: the system remembers the expected value for each hour and day of the week, accounts for daily and weekly cycles, maintenance windows, and the load's own trend.

Deviation is calculated not from an absolute number, but from how this particular service usually behaves.

Davis AI setup itself is not a switch.

You need a correct model of entities and relationships so that a failure of an underlying node does not raise a hundred notifications about dependent services, a baseline learning period, suppression rules during releases and maintenance, and alert profiles by service criticality. Let's break down how this is calibrated.

What calibration delivers: less noise and faster root cause

While thresholds are static, the flow of alerts grows faster than the infrastructure: the on-call engineer receives hundreds of signals per shift, stops reading them in full, and misses real degradation within the noise.

The cost of this mode is not only shift fatigue but also blurred priority: it is unclear which signal to raise first.

Dynatrace Davis AI builds a reference baseline from the actual behavior of services, not from a single number in a runbook. Calibration tunes this baseline to your traffic: daily and weekly peaks, release and maintenance windows, spikes from external integrations. Expected fluctuations stop being considered anomalies—the false positive rate drops severalfold.

The AI groups related incidents into one based on the shared service dependency: instead of several dozen alerts about different nodes, one arrives, indicating the root cause.

Time to detection is reduced to the first minutes of deviation, and time to root cause—from hours of searching through charts to reviewing a single dependency tree.

A separate effect is that an entire class of failures goes away: slow degradations that do not break through a fixed threshold (a quarter increase in response time at unchanged load) are now visible as an anomaly.

Detection accuracy grows without an increase in the number of alerts: fewer signals, but fuller problem coverage.

Analysis modes: comparing approaches to failure detection

The detection mode answers the question of who sets the boundary between normal and failure: an engineer manually, or an algorithm based on accumulated statistics.

This determines the false positive rate and the amount of work that goes into maintaining rules after every release or load increase.

Choosing a mode means choosing where manual rule calibration will remain and how much effort it will constantly require.

In practice, Dynatrace Davis AI is almost never used in a single mode: learning covers the background and seasonal fluctuations, manual boundaries hold critical paths, and relationship tagging determines whether the root cause appears in the very first signal.

Setup procedure: from audit to handover to operations

Davis AI setup is not "turn it on and forget it": the engine derives the root cause from the topology of services and event streams, so the result directly depends on data completeness and threshold accuracy.

The work order is structured so that by the time it is handed over to operations, the system does not create noise and actually points to the failure source.

1. Audit of rules and sources. We look at which systems already emit metrics and events, which alerts duplicate each other, and where thresholds are set "by eye" and produce false positives. The output is a list of sources with an assessment of their suitability.
2. Service dependency map. We link services to databases, queues, and external calls. Without a correct dependency map, cause-and-effect relationships are built incorrectly, and the affected service is assigned as the root cause rather than the one that broke.
3. Sensitivity calibration. We configure thresholds and averaging windows, suppress seasonal spikes and planned maintenance—for example, a nightly backup no longer raises an incident.
4. Pilot on a test environment. We take a limited group of services, run training scenarios, and verify that an alert arrives in the on-call channel with the correct root cause and clear context.
5. Validation on real incidents. We run through the history of past outages and compare whether the cause identified by the system matched the actual one. This is the most honest check of setup quality.
6. Routing and on-call schedules. We bind escalation rules to service owners so that an alert reaches someone who can fix it, not a general chat.
7. Handover and on-call training. We provide the runbook, walk through live examples of how to read an incident card and what to do on a false positive.

Output configurations, diagrams, and runbooks

Detection setup ends not with a report but with a working kit that the on-call shift is left with without us. Everything we calibrate in Dynatrace Davis AI is documented in configurations, diagrams, and runbooks tied to your services, teams, and areas of responsibility. The kit is designed to be read by an engineer at 3 a.m., not just by the person who configured it.

• Configuration package — thresholds and baselines for key services, rules for grouping related signals and suppressing duplicates, and linking triggers to owners.
• Service dependency map — a map by which the on-call engineer distinguishes the root cause from its effects and does not wake everyone for every noise.
• On-call runbook — criticality levels, target response time for each, escalation routes, and shift handover points.
• On-call instructions — step-by-step scenarios for typical triggers: what to check, what to restart, when to wake development.
• Rollback rules — how to return to previous thresholds and observation mode if the new logic produces excessive noise.
• Detection development plan — the order of connecting services and metrics, adding automated actions in subsequent cycles.
• Access and role matrix — who changes configurations, who only views, who approves changes.

Case study: how alert noise was removed on a distributed environment

An environment spanning several sites—services, databases, load balancers—ran on threshold alerts.

A rule triggered on a single latency spike, the on-call engineer received hundreds of notifications per shift, and some real degradations were lost in that noise: the service owner learned about a drop from users before monitoring.

What was changed

We took an inventory of dependencies and boundaries of the distributed environment and selected metrics suitable as baselines. We configured the topology—Dynatrace builds it from actual traffic, so links between services are not lost during releases.

We calibrated profiles for each service: where daily, where weekly, where spikes during peak hours are considered normal. Notifications were linked to service owners, and duplicates for the same root cause were suppressed.

What changed

The false positive rate fell from 60–70% to 10–15%: the model compares behavior with the service's own profile rather than a fixed threshold.

Time to root cause decreased from 40–60 minutes of manual investigation to 5–10—the on-call engineer immediately sees the problematic node and related events instead of the chain "alert → log → metric → neighboring service."

The number of missed degradations per month decreased from 8–10 to 1–2: weak drops that did not reach the threshold are now recorded as anomalies. The time freed from false signals is about a third of the on-call shift.

What to do if your own rules and dashboards are already deployed?

We do not tear anything down at the start. First, an inventory of rules: what triggered over the last quarter, what never triggered, and where thresholds duplicate each other.

Then—shadow mode: the new Dynatrace Davis AI environment runs in parallel but does not wake the on-call engineer, and its triggers are recorded separately and compared with yours. This creates a basis for a decision rather than a debate about preferences.

Monitoring dashboards are migrated, not redrawn: import preserves queries, breakdowns, and thresholds, and some of them simply stop being needed after switching to aggregated metrics.

The external environment does not have to be changed—chats, the incident management system, and on-call schedules receive notifications at the same addresses and in the usual format.

Rollout proceeds one zone at a time with a week of observation between steps. We keep rules as configuration in a version control system: they are disabled by profile or tag, the rollback point is the previous set within minutes, and the trigger history remains in place.

Common technical doubts before implementation

Before starting, they check not the product's capabilities but how it will live in your infrastructure: calibration timelines on your own data, behavior in the first days, replaceability of collection components, and the area of responsibility after handover to the team.

What happens in the first days after enabling?

At first we work in observation mode: signals are collected but not sent to on-call engineers. This allows the model's conclusions to be checked against real failure causes and noise to be removed before it reaches the on-call queue. We enable alerts in stages, starting with priority services.

Can collection components be replaced with ours?

Yes, this is a replaceable part. Existing agents and exporters are connected as sources; a separate collector is installed only where there is none of your own. The analytics core and accumulated observation history are not rebuilt during such a replacement.

What about support after handover?

We hand over configurations, correlation rules, and documentation, and train the on-call shift. After that—support: incident review and threshold fine-tuning based on actual operation.

We fix the boundaries of responsibility explicitly: we are responsible for signal quality and relationship logic, you are responsible for the availability of data sources.

Let's discuss the task: scope of work, plan, and timelines

To make the timeline estimate concrete rather than "from two weeks to six months," we build the implementation plan from your current state: which systems already emit telemetry, where there are coverage gaps, and what should be included in automated response.

Send us the inputs—we will come back with the scope of work by stage and the boundaries of responsibility for each.

• a list of services and environments that need to be covered by observability, with criticality marked;
• an infrastructure diagram: where components are deployed, which communication channels between environments are available;
• a list of event sources—what already writes to the monitoring system and what currently only goes to logs;
• a description of the current on-call process: how an alert currently arrives and who responds to it;
• 2-3 real cases where a failure was found later than desired—we use them to calibrate thresholds;
• access and permission separation requirements if the environments are isolated.

The first work cycle is always a pilot stage on one limited environment: we connect sources, configure basic event correlation, calibrate sensitivity to remove noise, and agree on an acceptance metric.

We record the result in measurable form: the share of failures detected before a user reports them, and the time from event to a ready alert with the cause indicated.

Then we scale to the remaining environments using the proven scheme—this is cheaper in terms of timelines than covering the entire fleet at once.

Describe the task in your own words and attach what you have at hand: a rough list of services and a couple of failure examples are enough for the first review. We will come back with a timeline estimate, a stage plan, and options for a pilot launch.