Why AI SOAR stalls without described scenarios
Response automation most often stalls not because of model quality, but because of the lack of a formal description of what the shift does manually. The platform is deployed, event sources are connected, but the share of incidents closed without a human remains low.
The analyst still opens an alert, goes to the console, cross-checks addresses, copies identifiers — and all this time the event stream grows faster than the shift can triage incidents.
The reason is that response playbooks are described “in general” rather than as a sequence of verifiable conditions.
If it is not recorded which source the event comes from, which fields are considered indicators of a trigger, and how one case differs from a similar one, automation cannot make a decision and goes into the “hand off to a human” branch.
Two analysts act differently in the same situation — which means there is no repeatable playbook, and there is nothing to train on such data.
A workable playbook looks different: the trigger is tied to a specific source in the event stream, enrichment conditions are defined, along with the list of permissible actions and the criterion by which a case is considered closed.
Then response automation closes typical cases end to end, and the on-duty shift gets involved only at decision points — where a choice is needed, not routine. We will break down how this is handled in practice.
The effect of automation: response speed and shift workload
The effect of automation is measured not by the number of playbooks, but by three values: time to first action on an incident, the share of routine removed from the analyst, and the volume of noise that never reaches the shift at all.
Before automation, a signal that arrived at the end of a shift waited for triage until morning — response time stretched to hours, and part of the event context was lost by the time it was triaged.
A response playbook triages an event in seconds: enriches it with asset data, checks it against history, filters out the obviously false, and performs the first containment action — account lockout or host isolation.
The analyst gets an assembled picture instead of a raw stream, and time to first action drops from tens of minutes to single digits. Platforms in the Cortex XSOAR class execute such playbooks as reproducible sets of steps — they can be versioned and rolled back.
Routine actions — exporting logs, cross-checking addresses and hashes, submitting to a sandbox, creating a ticket — go fully into automation.
What remains for the shift are decisions that require context and authority: roughly two or three out of five operations no longer occupy hands, and the analyst’s workload during peak hours levels out.
A separate class of failures associated with false positives also goes away. A rule that triggers on a single indicator is replaced by checking that several conditions match, and noisy rules are deprioritized instead of manually triaged.
The queue stops growing by the end of the shift, and shift handover happens from a stable state rather than inheriting a backlog of untriaged triggers.
AI SOAR: deployment options and their limitations
Deploying an automation loop rarely comes down to installing a single product. The choice is between a managed service from a vendor, an on-premises platform inside the perimeter, extending an already operating loop, and a hybrid.
The difference shows up not in the feature set, but in where incident data is stored, who is responsible for integration with security systems, and how quickly the response playbook can be changed.
Below are the alternatives as we see them in projects: by applicability, launch timelines, and the limitations that will have to be accepted consciously.
| Option | When it fits | Limitations |
|---|---|---|
| Managed service from a vendor | An automation loop is needed quickly, the team is small, and moving event data outside the perimeter is acceptable | Some incident data goes outside; playbook changes are limited by the platform’s capabilities |
| On-premises platform inside the perimeter | Data export is prohibited, there are regulatory requirements, and the event stream is predictable | Requires your own infrastructure and support; each security source is connected separately |
| Extending the current loop | Event collection and some automation playbooks are already working | Architectural limitations are inherited: data exchange bottlenecks move into the new layer |
| Hybrid: orchestration inside, analytics outside | Control over actions in the infrastructure is needed, along with external analytics on anonymized data | Harder to support: two loops, two access-control models, rule synchronization |
The decision is fixed by limitations, not by strengths: a managed service runs into data storage requirements, an on-premises platform into the amount of support, and extending the current loop into the quality of existing integrations.
That is why we start the review with your security landscape and only then propose a deployment option.
Implementation stages: from process audit to handover to operations
Automation is justified where operations repeat and event sources are clear. That is why AI SOAR starts with a process audit: unless you record which alerts the shift triages manually and how many steps each one takes, automation will cement the existing disorder. Below is the sequence of work we go through on a project.
- Process audit. We collect event sources and current manual operations: how alerts arrive, how they are classified, where time is lost. The output is a flow map and a list of operations suitable for automation.
- Response design. For each incident class, we describe the branch: data enrichment, host isolation, account lockout, notification of the owner. We agree on boundaries — where the system makes the decision and where manual review is mandatory.
- Test loop. We deploy an isolated environment with anonymized data that mirrors the production topology: an error in a playbook does not reach production, and rollback takes minutes.
- Playbook configuration. We connect event sources, configure correlation rules and trigger thresholds, and assemble processing branches. We filter out false positives during triage, not on the shift.
- Validation on real incidents. We run the playbooks against historical logs and the live stream, and measure the share of alerts closed without a human and the response time. We close any gaps found before handover.
- Training and handover to operations. We provide regulations and instructions for non-standard cases and train the shift on the test loop. After that comes support and refinement of playbooks based on accumulated statistics.
What we hand over: configurations, playbooks, regulations, and documentation
A SOC automation loop lives in operations, not in a demo: the on-duty engineer must understand where an event came from, why the playbook acted the way it did, and what to do if it did not trigger.
That is why we hand over not only a working system, but also a package that can be used to run it further without our involvement.
- Configurations — an export of correlation, routing, and event prioritization rules with comments: you can see what each rule is responsible for and what will change if it is disabled.
- Integration schemes — a map of event sources and external systems: which protocols are used, where credentials are stored, and what retries are configured when a source is unavailable.
- Regulations for the shift — step-by-step instructions for typical situations: initial triage, escalation, host isolation, incident closure, and actions for a false positive.
- Response playbooks — automations with triggers, conditions, and boundaries: what the system does itself and where human review is mandatory.
- Access policies — a role matrix: who sees events, who changes rules, who confirms automatic actions, and how rights are granted and revoked.
- Documentation — a description of the architecture, data flows, failure points, and the procedure for updating components.
- Validation procedure — a set of test events the shift uses to confirm playbook performance after changes, and the criteria under which an update is rolled back.
- Change log — the history of rule edits with author and reason: incident review relies on facts, not memory.
Case study: how we reduced the response time of the on-duty shift
Before implementation: the on-duty analyst as a bottleneck
At the start of the implementation case, the shift received around 4,000 triggers per day from SIEM — of varying reliability and with no overall prioritization.
The analyst manually matched the event to assets and pulled context from email and tickets, and initial triage took 35–45 minutes, while during night hours the queue simply waited until morning.
Shift handover relied on a chat recap: context was lost, and sometimes the same incident was triaged twice.
What we changed
We built a library of response playbooks based on the shared taxonomy of MITRE ATT&CK tactics: event classification, automatic enrichment with asset and account data, host isolation, and account lockout according to pre-agreed thresholds. Enrichment runs in parallel — while the analyst opens the ticket, the context is already assembled.
Every action of the automation is written to a log with the playbook and input data indicated, so incident review does not turn into archaeology. Shift handover was moved to a checklist: open incidents, deferred decisions, triggered exceptions.
Before and after metrics
Average response time dropped from 42 to 7 minutes, and the median closure time for similar incidents fell from 6 hours to 40 minutes. The share of events closed without human involvement grew to two-thirds; manual actions per incident decreased by a factor of 4. The night shift workload leveled out: the queue stopped accumulating by morning.
Where it was harder than expected
Two areas required separate iterations. The first was sources with non-standard log formats: some had to be brought to a common form on the collection side, otherwise the classifier made mistakes on similar events.
The second was analysts’ trust in automation: at first, some playbooks ran in “suggest an action” mode rather than “execute,” and only after accumulated statistics did we remove step-by-step confirmation.
What to do if your own response automation is already deployed?
Your own automation is already deployed — that is not a reason to start from scratch, but a ready foundation: some playbooks describe your processes more precisely than any out-of-the-box set.
That is why we do not shut down the existing loop but inherit it — we connect to the same event sources, preserve alert routing, service account permissions, and incident numbering so that the response history is not broken.
Next comes a playbook review: we take inventory of what works now. We look at which rules actually trigger, what the false-positive rate is, where the analyst still finishes the work manually, and which playbooks duplicate each other in meaning.
Every rule gets a verdict — leave as is, rewrite the logic, or archive it. In practice, a noticeable portion of custom rules either never triggered once or conflicts with neighboring rules and suppresses useful alerts.
Playbook migration is staged and without shift downtime. We launch new rules in shadow mode: they see the same events but do not perform actions — we compare what the automation proposed and what the on-duty analyst did.
A loop on Shuffle or on custom scripts is handled the same way: the logic is transferred branch by branch, preserving escalation points and with the ability to return to the previous rule with a single switch.
Inheriting the loop ends with handover to support, not with launch.
We establish the regulation: who adds a playbook, on which test events it is validated, and by which metrics degradation is visible — time to first automatic action, share of incidents closed without a human, number of automation rollbacks.
After that, the loop evolves together with your processes rather than remaining a frozen configuration.
Integrations with existing systems and false positives
SOAR brings no benefit on its own — it works exactly as well as event sources provide data and as cleanly as they are brought to a common schema.
At the start of a project, the same technical doubts almost always surface: what to do with systems that have no machine access, how to maintain trigger accuracy after the first weeks of operation, and who updates playbooks when the infrastructure changes.
What to do with event sources that have no API?
We connect available systems directly and normalize events into a common schema. For closed solutions, we deploy a collector agent for export or parse text logs — the source joins the common stream instead of remaining aside.
For options with no machine access at all, we move to semi-manual mode: the analyst receives a ready playbook and confirms the action instead of assembling context manually.
How do you reduce the share of false positives?
Every trigger is enriched with context: who and from which host, what happened on that host earlier, and whether the pattern repeats. A single signal remains an observation; an automatic action is launched when several indicators match.
For the first weeks, the system runs in accumulation mode: we compare its decisions with analysts’ conclusions and tune thresholds until the false-positive share drops to an agreed level.
What is included in support and how are playbooks updated?
Playbooks are versioned, and each has a regression set of events — a change does not go into operation until it passes it completely. When the infrastructure changes, we add new sources and rules and decommission old ones rather than accumulating them.
Rollback to the previous revision takes minutes: if an updated playbook behaves differently, we restore the previous logic without stopping the shift.
Let’s discuss your task and come back with an estimate and plan
We start the conversation with the task, not with a presentation: which incident classes need to be closed automatically, how many events per day enter the loop, and which actions the analyst performs manually most often.
To start, all we need from you is a list of event sources, ways to integrate with them, current response regulations, and two or three reviewed incidents with step-by-step timing.
Then we clarify what is rarely described in documentation: completeness of event normalization, quality of asset and account directories, and separation of rights for automatic actions.
If integrations are heterogeneous, we start with an audit of the current loop — it shows which playbooks can be automated immediately and which will run into a lack of data.
- a map of event sources and integration readiness, with explicit data gaps;
- a list of response playbooks sorted by trigger frequency and labor effort;
- a stage-by-stage work plan — from loop audit to the first automated playbooks;
- timeline estimates for each stage and the team composition for it;
- a description of the pilot stage: which playbooks we take first and how we measure the result;
- requirements for access, test data, and environments so the pilot does not stall in approvals;
- acceptance criteria: the share of incidents closed without analyst involvement and the response time.
Describe your task in free form — we will come back with a timeline estimate and work plan.
